You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Ruben Laguna <ru...@gmail.com> on 2010/04/07 22:35:50 UTC
IndexWriter memory leak?
Hi,
It seems like my IndexWriter after commiting and optimizing has a retained
size of 140Mb. See [1] for a screenshot of the heapdump analysis done with
Eclipse MAT.
Of those 140MB 67MB are retained by
analyzer.tokenStreams.hardRefs.table.HashMap$Entry.value.tokenStream.scanner.zzBuffer
why is this? Is it a memory leak? or did I something wrong during the
indxing? (BTW, I'm indexing document which contains Fields(xxxx,Reader) and
those Reader are wrappers around Tika.parse(xxxx) Readers. I get a lot
IOExceptions from tika readers and the wrapper maps the exceptions to EOF so
Lucene doesn't see the exception).
...and 73MB of the 140MB are retained by docWriter see [2]. It looks like
the Field objects in the
array docWriter.threadStates[0].consumer.fieldHash[1].fields[xxxx] are
holding references to the Readers. Those reader instances are actually
closed after IndexWriter.updateDocument. Each one of those Readers retains
1MB. The question is why IndexWriter holds references to those Readers after
the Documents have been indexed.
[1] http://img.skitch.com/20100407-1183815yiausisg73u9wfgscsj.jpg
[2] http://img.skitch.com/20100407-b86irkp7e4uif2wq1dd4t899qb.jpg
--
/Rubén
RE: IndexWriter memory leak?
Posted by Uwe Schindler <uw...@thetaphi.de>.
I responded, because the mentioned issue will change the whole class structure in the standard package, so any patch would get outdated, soon. So it’s the best to add it directly there.
But if you try out, if it works, that’s fine. The fix would be in 3.1 so if you need to fix your 3.0.1 version, you have to patch manually :-)
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: Ruben Laguna [mailto:ruben.laguna@gmail.com]
> Sent: Thursday, April 08, 2010 12:05 PM
> To: java-user@lucene.apache.org
> Subject: Re: IndexWriter memory leak?
>
> That was fast! I was already writting a patch... just to see if it
> works.
>
> On Thu, Apr 8, 2010 at 12:02 PM, Uwe Schindler <uw...@thetaphi.de> wrote:
>
> > Hi Shai, hi Ruben,
> >
> > I will take care of this in
> > https://issues.apache.org/jira/browse/LUCENE-2074 where some parts of
> the
> > Tokenizer impl are rewritten.
> >
> > -----
> > Uwe Schindler
> > H.-H.-Meier-Allee 63, D-28213 Bremen
> > http://www.thetaphi.de
> > eMail: uwe@thetaphi.de
> >
> >
> > > -----Original Message-----
> > > From: Ruben Laguna [mailto:ruben.laguna@gmail.com]
> > > Sent: Thursday, April 08, 2010 11:51 AM
> > > To: java-user@lucene.apache.org
> > > Subject: Re: IndexWriter memory leak?
> > >
> > > I was investigating this a little further and in the JFlex mailing
> list
> > > I
> > > found [1]
> > >
> > > I don't know much about flex / JFlex but it seems that this guy
> resets
> > > the
> > > zzBuffer to 16384 or less when setting the input for the lexer
> > >
> > >
> > > Quoted from shef <sh...@ya...>
> > >
> > >
> > > I set
> > >
> > > %buffer 0
> > >
> > > in the options section, and then added this method to the lexer:
> > >
> > > /**
> > > * Set the input for the lexer. The size parameter really
> speeds
> > > things up,
> > > * because by default, the lexer allocates an internal buffer
> of
> > > 16k. For
> > > * most strings, this is unnecessarily large. If the size param
> is
> > > 0 or greater
> > > * than 16k, then the buffer is set to 16k. If the size param
> is
> > > smaller, then
> > > * the buf will be set to the exact size.
> > > * @param r the reader that provides the data
> > > * @param the size of the data in the reader.
> > > */
> > > public void reset(Reader r, int size) {
> > > if (size == 0 || size > 16384)
> > > size = 16384;
> > > zzBuffer = new char[size];
> > > yyreset(r);
> > > }
> > >
> > >
> > > So maybe there is a way to trim the zzBuffer this way (?).
> > >
> > > BTW, I will try to find out which is the "big token" in my dataset
> this
> > > afternoon. Thanks for the help.
> > >
> > > I actually workaround this memory problem for the time being by
> > > wrapping the
> > > IndexWriter in a class that periodically closes the IndexWriter and
> > > creates
> > > a new one, allowing the old to be GCed, but I would be really good
> if
> > > either
> > > JFlex or Lucene can take care of this zzBuffer going berserk.
> > >
> > >
> > > Again thanks for the quick response. /Rubén
> > >
> > >
> > > [1]
> > >
> https://sourceforge.net/mailarchive/message.php?msg_id=444070.38422.qm@
> > > web38901.mail.mud.yahoo.com
> > >
> > > On Thu, Apr 8, 2010 at 11:32 AM, Shai Erera <se...@gmail.com>
> wrote:
> > >
> > > > If we could change the Flex file so that yyreset(Reader) would
> check
> > > the
> > > > size of zzBuffer, we could trim it when it gets too big. But I
> don't
> > > think
> > > > we have such control when writing the flex syntax ... yyreset is
> > > generated
> > > > by JFlex and that's the only place I can think of to trim the
> buffer
> > > down
> > > > when it exceeds a predefined threshold ....
> > > >
> > > > Maybe what we can do is create our own method which will be
> called by
> > > > StandardTokenizer after yyreset is called, something like
> > > > trimBufferIfTooBig(int threshold) which will reallocate zzBuffer
> if
> > > it
> > > > exceeded the threshold. We can decide on a reasonable 64K
> threshold
> > > or
> > > > something, or simply always cut back to 16 KB. As far as I
> > > understand, that
> > > > buffer should never grow that much. I.e. in zzRefill, which is
> the
> > > only
> > > > place where the buffer gets resized, there is an attempt to first
> > > move back
> > > > characters that were already consumed and only then allocate a
> bigger
> > > > buffer. Which means only if there is a token whose size is larger
> > > than 16KB
> > > > (!?), will this buffer get expanded.
> > > >
> > > > A trimBuffer method might not be that bad .. as a protective
> measure.
> > > What
> > > > do you think? Of course, JFlex can fix it on their own ... but
> until
> > > that
> > > > happens ...
> > > >
> > > > Shai
> > > >
> > > > On Thu, Apr 8, 2010 at 10:35 AM, Uwe Schindler <uw...@thetaphi.de>
> > > wrote:
> > > >
> > > > > > I would like to identify also the problematic document I have
> > > 10000 so,
> > > > > > what
> > > > > > would be the best way of identifying the one that it making
> > > zzBuffer to
> > > > > > grow
> > > > > > without control?
> > > > >
> > > > > Dont index your documents, but instead pass them directly to
> the
> > > analyzer
> > > > > and consume the tokenstream manually. Then visit
> > > > TermAttribute.termLength()
> > > > > for each Token.
> > > > >
> > > > >
> > > > > ---------------------------------------------------------------
> ----
> > > --
> > > > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > > > > For additional commands, e-mail: java-user-
> help@lucene.apache.org
> > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > /Rubén
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
>
>
> --
> /Rubén
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: IndexWriter memory leak?
Posted by Ruben Laguna <ru...@gmail.com>.
That was fast! I was already writting a patch... just to see if it works.
On Thu, Apr 8, 2010 at 12:02 PM, Uwe Schindler <uw...@thetaphi.de> wrote:
> Hi Shai, hi Ruben,
>
> I will take care of this in
> https://issues.apache.org/jira/browse/LUCENE-2074 where some parts of the
> Tokenizer impl are rewritten.
>
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: uwe@thetaphi.de
>
>
> > -----Original Message-----
> > From: Ruben Laguna [mailto:ruben.laguna@gmail.com]
> > Sent: Thursday, April 08, 2010 11:51 AM
> > To: java-user@lucene.apache.org
> > Subject: Re: IndexWriter memory leak?
> >
> > I was investigating this a little further and in the JFlex mailing list
> > I
> > found [1]
> >
> > I don't know much about flex / JFlex but it seems that this guy resets
> > the
> > zzBuffer to 16384 or less when setting the input for the lexer
> >
> >
> > Quoted from shef <sh...@ya...>
> >
> >
> > I set
> >
> > %buffer 0
> >
> > in the options section, and then added this method to the lexer:
> >
> > /**
> > * Set the input for the lexer. The size parameter really speeds
> > things up,
> > * because by default, the lexer allocates an internal buffer of
> > 16k. For
> > * most strings, this is unnecessarily large. If the size param is
> > 0 or greater
> > * than 16k, then the buffer is set to 16k. If the size param is
> > smaller, then
> > * the buf will be set to the exact size.
> > * @param r the reader that provides the data
> > * @param the size of the data in the reader.
> > */
> > public void reset(Reader r, int size) {
> > if (size == 0 || size > 16384)
> > size = 16384;
> > zzBuffer = new char[size];
> > yyreset(r);
> > }
> >
> >
> > So maybe there is a way to trim the zzBuffer this way (?).
> >
> > BTW, I will try to find out which is the "big token" in my dataset this
> > afternoon. Thanks for the help.
> >
> > I actually workaround this memory problem for the time being by
> > wrapping the
> > IndexWriter in a class that periodically closes the IndexWriter and
> > creates
> > a new one, allowing the old to be GCed, but I would be really good if
> > either
> > JFlex or Lucene can take care of this zzBuffer going berserk.
> >
> >
> > Again thanks for the quick response. /Rubén
> >
> >
> > [1]
> > https://sourceforge.net/mailarchive/message.php?msg_id=444070.38422.qm@
> > web38901.mail.mud.yahoo.com
> >
> > On Thu, Apr 8, 2010 at 11:32 AM, Shai Erera <se...@gmail.com> wrote:
> >
> > > If we could change the Flex file so that yyreset(Reader) would check
> > the
> > > size of zzBuffer, we could trim it when it gets too big. But I don't
> > think
> > > we have such control when writing the flex syntax ... yyreset is
> > generated
> > > by JFlex and that's the only place I can think of to trim the buffer
> > down
> > > when it exceeds a predefined threshold ....
> > >
> > > Maybe what we can do is create our own method which will be called by
> > > StandardTokenizer after yyreset is called, something like
> > > trimBufferIfTooBig(int threshold) which will reallocate zzBuffer if
> > it
> > > exceeded the threshold. We can decide on a reasonable 64K threshold
> > or
> > > something, or simply always cut back to 16 KB. As far as I
> > understand, that
> > > buffer should never grow that much. I.e. in zzRefill, which is the
> > only
> > > place where the buffer gets resized, there is an attempt to first
> > move back
> > > characters that were already consumed and only then allocate a bigger
> > > buffer. Which means only if there is a token whose size is larger
> > than 16KB
> > > (!?), will this buffer get expanded.
> > >
> > > A trimBuffer method might not be that bad .. as a protective measure.
> > What
> > > do you think? Of course, JFlex can fix it on their own ... but until
> > that
> > > happens ...
> > >
> > > Shai
> > >
> > > On Thu, Apr 8, 2010 at 10:35 AM, Uwe Schindler <uw...@thetaphi.de>
> > wrote:
> > >
> > > > > I would like to identify also the problematic document I have
> > 10000 so,
> > > > > what
> > > > > would be the best way of identifying the one that it making
> > zzBuffer to
> > > > > grow
> > > > > without control?
> > > >
> > > > Dont index your documents, but instead pass them directly to the
> > analyzer
> > > > and consume the tokenstream manually. Then visit
> > > TermAttribute.termLength()
> > > > for each Token.
> > > >
> > > >
> > > > -------------------------------------------------------------------
> > --
> > > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > > > For additional commands, e-mail: java-user-help@lucene.apache.org
> > > >
> > > >
> > >
> >
> >
> >
> > --
> > /Rubén
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
--
/Rubén
RE: IndexWriter memory leak?
Posted by Uwe Schindler <uw...@thetaphi.de>.
Hi Shai, hi Ruben,
I will take care of this in https://issues.apache.org/jira/browse/LUCENE-2074 where some parts of the Tokenizer impl are rewritten.
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: Ruben Laguna [mailto:ruben.laguna@gmail.com]
> Sent: Thursday, April 08, 2010 11:51 AM
> To: java-user@lucene.apache.org
> Subject: Re: IndexWriter memory leak?
>
> I was investigating this a little further and in the JFlex mailing list
> I
> found [1]
>
> I don't know much about flex / JFlex but it seems that this guy resets
> the
> zzBuffer to 16384 or less when setting the input for the lexer
>
>
> Quoted from shef <sh...@ya...>
>
>
> I set
>
> %buffer 0
>
> in the options section, and then added this method to the lexer:
>
> /**
> * Set the input for the lexer. The size parameter really speeds
> things up,
> * because by default, the lexer allocates an internal buffer of
> 16k. For
> * most strings, this is unnecessarily large. If the size param is
> 0 or greater
> * than 16k, then the buffer is set to 16k. If the size param is
> smaller, then
> * the buf will be set to the exact size.
> * @param r the reader that provides the data
> * @param the size of the data in the reader.
> */
> public void reset(Reader r, int size) {
> if (size == 0 || size > 16384)
> size = 16384;
> zzBuffer = new char[size];
> yyreset(r);
> }
>
>
> So maybe there is a way to trim the zzBuffer this way (?).
>
> BTW, I will try to find out which is the "big token" in my dataset this
> afternoon. Thanks for the help.
>
> I actually workaround this memory problem for the time being by
> wrapping the
> IndexWriter in a class that periodically closes the IndexWriter and
> creates
> a new one, allowing the old to be GCed, but I would be really good if
> either
> JFlex or Lucene can take care of this zzBuffer going berserk.
>
>
> Again thanks for the quick response. /Rubén
>
>
> [1]
> https://sourceforge.net/mailarchive/message.php?msg_id=444070.38422.qm@
> web38901.mail.mud.yahoo.com
>
> On Thu, Apr 8, 2010 at 11:32 AM, Shai Erera <se...@gmail.com> wrote:
>
> > If we could change the Flex file so that yyreset(Reader) would check
> the
> > size of zzBuffer, we could trim it when it gets too big. But I don't
> think
> > we have such control when writing the flex syntax ... yyreset is
> generated
> > by JFlex and that's the only place I can think of to trim the buffer
> down
> > when it exceeds a predefined threshold ....
> >
> > Maybe what we can do is create our own method which will be called by
> > StandardTokenizer after yyreset is called, something like
> > trimBufferIfTooBig(int threshold) which will reallocate zzBuffer if
> it
> > exceeded the threshold. We can decide on a reasonable 64K threshold
> or
> > something, or simply always cut back to 16 KB. As far as I
> understand, that
> > buffer should never grow that much. I.e. in zzRefill, which is the
> only
> > place where the buffer gets resized, there is an attempt to first
> move back
> > characters that were already consumed and only then allocate a bigger
> > buffer. Which means only if there is a token whose size is larger
> than 16KB
> > (!?), will this buffer get expanded.
> >
> > A trimBuffer method might not be that bad .. as a protective measure.
> What
> > do you think? Of course, JFlex can fix it on their own ... but until
> that
> > happens ...
> >
> > Shai
> >
> > On Thu, Apr 8, 2010 at 10:35 AM, Uwe Schindler <uw...@thetaphi.de>
> wrote:
> >
> > > > I would like to identify also the problematic document I have
> 10000 so,
> > > > what
> > > > would be the best way of identifying the one that it making
> zzBuffer to
> > > > grow
> > > > without control?
> > >
> > > Dont index your documents, but instead pass them directly to the
> analyzer
> > > and consume the tokenstream manually. Then visit
> > TermAttribute.termLength()
> > > for each Token.
> > >
> > >
> > > -------------------------------------------------------------------
> --
> > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > > For additional commands, e-mail: java-user-help@lucene.apache.org
> > >
> > >
> >
>
>
>
> --
> /Rubén
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: IndexWriter memory leak?
Posted by Michael McCandless <lu...@mikemccandless.com>.
Thank you for isolating & raising the issue!!
Mike
On Sat, Apr 10, 2010 at 1:28 AM, Ruben Laguna <ru...@gmail.com> wrote:
> I just tried the changes that you commited it works beatifully. The readers
> are GCed. Thanks for both LUCENE-2387 and LUCENE-2384. Those make a big
> difference in my app!
>
> On Fri, Apr 9, 2010 at 12:32 PM, Michael McCandless
> <lu...@mikemccandless.com> wrote:
>>
>> I agree IW should not hold refs to the Field instances from the last
>> doc indexed... I put a patch on LUCENE-2387 to null the reference as
>> we go. Can you confirm this lets GC reclaim?
>>
>> Mike
>>
>> On Fri, Apr 9, 2010 at 12:54 AM, Ruben Laguna <ru...@gmail.com>
>> wrote:
>> > But the Readers I'm talking about are not held by the Tokenizer (at
>> > least
>> > not *only* by it), these are held by the DocFieldProccessorPerThread....
>> >
>> > IndexWriter -> DocumentsWriter -> DocumentsWriterThreadState ->
>> > DocFieldProcessorPerThread -> DocFieldProcessorPerField -> Fieldable ->
>> > Field (fieldsData)
>> >
>> > and it's not only one Reader there are several one (one per thread I
>> > suppose, in my heapdump there is 25 Reader that should have been GCed
>> > otherwise).
>> >
>> > Best regards/Ruben
>> > On Thu, Apr 8, 2010 at 11:49 PM, Uwe Schindler <uw...@thetaphi.de> wrote:
>> >
>> >> There is one possibility, that could be fixed:
>> >>
>> >> As Tokenizers are reused, the analyzer holds a reference to the last
>> >> used
>> >> Reader. The easy fix would be to unset the Reader in Tokenizer.close().
>> >> If
>> >> this is the case for you, that may be easy to do. So Tokenizer.close()
>> >> looks
>> >> like this:
>> >>
>> >> /** By default, closes the input Reader. */
>> >> @Override
>> >> public void close() throws IOException {
>> >> input.close();
>> >> input = null; // <-- new!
>> >> }
>> >>
>> >> Any comments from other committers?
>> >>
>> >> -----
>> >> Uwe Schindler
>> >> H.-H.-Meier-Allee 63, D-28213 Bremen
>> >> http://www.thetaphi.de
>> >> eMail: uwe@thetaphi.de
>> >>
>> >>
>> >> > -----Original Message-----
>> >> > From: Ruben Laguna [mailto:ruben.laguna@gmail.com]
>> >> > Sent: Thursday, April 08, 2010 2:50 PM
>> >> > To: java-user@lucene.apache.org
>> >> > Subject: Re: IndexWriter memory leak?
>> >> >
>> >> > I will double check in the afternoon the heapdump.hprof. But I think
>> >> > that
>> >> > *some* readers are indeed held by
>> >> > docWriter.threadStates[0].consumer.fieldHash[1].fields[xxxx],
>> >> > as shown in [1] (this heapdump contains only live objects). The
>> >> > heapdump
>> >> > was taken after IndexWriter.commit() /IndexWriter.optimize() and all
>> >> > the
>> >> > Documents were already indexed and GCed (I will double check).
>> >> >
>> >> > So that would mean that the Reader is retained in memory by the
>> >> > following
>> >> > chaing of references,
>> >> >
>> >> > DocumentsWriter -> DocumentsWriterThreadState ->
>> >> > DocFieldProcessorPerThread
>> >> > -> DocFieldProcessorPerField -> Fieldable -> Field (fieldsData)
>> >> >
>> >> > I'll double check with Eclipse MAT as I said that this chain is
>> >> > actually
>> >> > made of hard references only (no SoftReferences,WeakReferences, etc).
>> >> > I
>> >> > will
>> >> > also double check also that there is no "live" Document that is
>> >> > referencing
>> >> > the Reader via the Field.
>> >> >
>> >> >
>> >> > [1] http://img.skitch.com/20100407-b86irkp7e4uif2wq1dd4t899qb.jpg
>> >> >
>> >> > On Thu, Apr 8, 2010 at 2:16 PM, Uwe Schindler <uw...@thetaphi.de>
>> >> > wrote:
>> >> >
>> >> > > Readers are not held. If you indexed the document and gced the
>> >> > document
>> >> > > instance they readers are gone.
>> >> > >
>> >> > > -----
>> >> > > Uwe Schindler
>> >> > > H.-H.-Meier-Allee 63, D-28213 Bremen
>> >> > > http://www.thetaphi.de
>> >> > > eMail: uwe@thetaphi.de
>> >> > >
>> >> > >
>> >> > > > -----Original Message-----
>> >> > > > From: Ruben Laguna [mailto:ruben.laguna@gmail.com]
>> >> > > > Sent: Thursday, April 08, 2010 1:28 PM
>> >> > > > To: java-user@lucene.apache.org
>> >> > > > Subject: Re: IndexWriter memory leak?
>> >> > > >
>> >> > > > Now that the zzBuffer issue is solved...
>> >> > > >
>> >> > > > what about the references to the Readers held by docWriter.
>> >> > > > Tika´s
>> >> > > > ParsingReaders are quite heavyweight so retaining those in memory
>> >> > > > unnecesarily is also a "hidden" memory leak. Should I open a bug
>> >> > report
>> >> > > > on
>> >> > > > that one?
>> >> > > >
>> >> > > > /Rubén
>> >> > > >
>> >> > > > On Thu, Apr 8, 2010 at 12:11 PM, Shai Erera <se...@gmail.com>
>> >> > wrote:
>> >> > > >
>> >> > > > > Guess we were replying at the same time :).
>> >> > > > >
>> >> > > > > On Thu, Apr 8, 2010 at 1:04 PM, Uwe Schindler <uw...@thetaphi.de>
>> >> > > > wrote:
>> >> > > > >
>> >> > > > > > I already answered, that I will take care of this!
>> >> > > > > >
>> >> > > > > > Uwe
>> >> > > > > >
>> >> > > > > > -----
>> >> > > > > > Uwe Schindler
>> >> > > > > > H.-H.-Meier-Allee 63, D-28213 Bremen
>> >> > > > > > http://www.thetaphi.de
>> >> > > > > > eMail: uwe@thetaphi.de
>> >> > > > > >
>> >> > > > > >
>> >> > > > > > > -----Original Message-----
>> >> > > > > > > From: Shai Erera [mailto:serera@gmail.com]
>> >> > > > > > > Sent: Thursday, April 08, 2010 12:00 PM
>> >> > > > > > > To: java-user@lucene.apache.org
>> >> > > > > > > Subject: Re: IndexWriter memory leak?
>> >> > > > > > >
>> >> > > > > > > Yes, that's the trimBuffer version I was thinking about,
>> >> > > > > > > only
>> >> > > > this guy
>> >> > > > > > > created a reset(Reader, int) and does both ops (resetting +
>> >> > trim)
>> >> > > > in
>> >> > > > > > > one
>> >> > > > > > > method call. More convenient. Can you please open an issue
>> >> > > > > > > to
>> >> > > > track
>> >> > > > > > > that?
>> >> > > > > > > People will have a chance to comment on whether we (Lucene)
>> >> > > > should
>> >> > > > > > > handle
>> >> > > > > > > that, or it should be a JFlex fix. Based on the number of
>> >> > replies
>> >> > > > this
>> >> > > > > > > guy
>> >> > > > > > > received (0 !), I doubt JFlex would consider it a problem.
>> >> > But we
>> >> > > > can
>> >> > > > > > > do
>> >> > > > > > > some small service to our users base by protecting against
>> >> > such
>> >> > > > > > > problems.
>> >> > > > > > >
>> >> > > > > > > And while you're opening the issue, if you want to take a
>> >> > stab at
>> >> > > > > > > fixing it
>> >> > > > > > > and post a patch, it'd be great :).
>> >> > > > > > >
>> >> > > > > > > Shai
>> >> > > > > > >
>> >> > > > > > > On Thu, Apr 8, 2010 at 12:51 PM, Ruben Laguna
>> >> > > > > > > <ru...@gmail.com>wrote:
>> >> > > > > > >
>> >> > > > > > > > I was investigating this a little further and in the
>> >> > > > > > > > JFlex
>> >> > > > mailing
>> >> > > > > > > list I
>> >> > > > > > > > found [1]
>> >> > > > > > > >
>> >> > > > > > > > I don't know much about flex / JFlex but it seems that
>> >> > > > > > > > this
>> >> > guy
>> >> > > > > > > resets the
>> >> > > > > > > > zzBuffer to 16384 or less when setting the input for the
>> >> > lexer
>> >> > > > > > > >
>> >> > > > > > > >
>> >> > > > > > > > Quoted from shef <sh...@ya...>
>> >> > > > > > > >
>> >> > > > > > > >
>> >> > > > > > > > I set
>> >> > > > > > > >
>> >> > > > > > > > %buffer 0
>> >> > > > > > > >
>> >> > > > > > > > in the options section, and then added this method to the
>> >> > > > lexer:
>> >> > > > > > > >
>> >> > > > > > > > /**
>> >> > > > > > > > * Set the input for the lexer. The size parameter
>> >> > really
>> >> > > > speeds
>> >> > > > > > > things
>> >> > > > > > > > up,
>> >> > > > > > > > * because by default, the lexer allocates an internal
>> >> > > > buffer of
>> >> > > > > > > 16k.
>> >> > > > > > > > For
>> >> > > > > > > > * most strings, this is unnecessarily large. If the
>> >> > size
>> >> > > > param is
>> >> > > > > > > > 0 or greater
>> >> > > > > > > > * than 16k, then the buffer is set to 16k. If the
>> >> > > > > > > > size
>> >> > > > param is
>> >> > > > > > > > smaller, then
>> >> > > > > > > > * the buf will be set to the exact size.
>> >> > > > > > > > * @param r the reader that provides the data
>> >> > > > > > > > * @param the size of the data in the reader.
>> >> > > > > > > > */
>> >> > > > > > > > public void reset(Reader r, int size) {
>> >> > > > > > > > if (size == 0 || size > 16384)
>> >> > > > > > > > size = 16384;
>> >> > > > > > > > zzBuffer = new char[size];
>> >> > > > > > > > yyreset(r);
>> >> > > > > > > > }
>> >> > > > > > > >
>> >> > > > > > > >
>> >> > > > > > > > So maybe there is a way to trim the zzBuffer this way
>> >> > > > > > > > (?).
>> >> > > > > > > >
>> >> > > > > > > > BTW, I will try to find out which is the "big token" in
>> >> > > > > > > > my
>> >> > > > dataset
>> >> > > > > > > this
>> >> > > > > > > > afternoon. Thanks for the help.
>> >> > > > > > > >
>> >> > > > > > > > I actually workaround this memory problem for the time
>> >> > being by
>> >> > > > > > > wrapping
>> >> > > > > > > > the
>> >> > > > > > > > IndexWriter in a class that periodically closes the
>> >> > IndexWriter
>> >> > > > and
>> >> > > > > > > creates
>> >> > > > > > > > a new one, allowing the old to be GCed, but I would be
>> >> > really
>> >> > > > good if
>> >> > > > > > > > either
>> >> > > > > > > > JFlex or Lucene can take care of this zzBuffer going
>> >> > berserk.
>> >> > > > > > > >
>> >> > > > > > > >
>> >> > > > > > > > Again thanks for the quick response. /Rubén
>> >> > > > > > > >
>> >> > > > > > > >
>> >> > > > > > > > [1]
>> >> > > > > > > >
>> >> > > > > > > >
>> >> > > > > > >
>> >> > > > >
>> >> > > >
>> >> >
>> >> > https://sourceforge.net/mailarchive/message.php?msg_id=444070.38422.qm@
>> >> > > > > > > web38901.mail.mud.yahoo.com
>> >> > > > > > > >
>> >> > > > > > > > On Thu, Apr 8, 2010 at 11:32 AM, Shai Erera
>> >> > <se...@gmail.com>
>> >> > > > > wrote:
>> >> > > > > > > >
>> >> > > > > > > > > If we could change the Flex file so that
>> >> > > > > > > > > yyreset(Reader)
>> >> > > > would
>> >> > > > > > > check the
>> >> > > > > > > > > size of zzBuffer, we could trim it when it gets too
>> >> > > > > > > > > big.
>> >> > But
>> >> > > > I
>> >> > > > > > > don't
>> >> > > > > > > > think
>> >> > > > > > > > > we have such control when writing the flex syntax ...
>> >> > yyreset
>> >> > > > is
>> >> > > > > > > > generated
>> >> > > > > > > > > by JFlex and that's the only place I can think of to
>> >> > > > > > > > > trim
>> >> > the
>> >> > > > > > > buffer down
>> >> > > > > > > > > when it exceeds a predefined threshold ....
>> >> > > > > > > > >
>> >> > > > > > > > > Maybe what we can do is create our own method which
>> >> > > > > > > > > will
>> >> > be
>> >> > > > called
>> >> > > > > > > by
>> >> > > > > > > > > StandardTokenizer after yyreset is called, something
>> >> > > > > > > > > like
>> >> > > > > > > > > trimBufferIfTooBig(int threshold) which will reallocate
>> >> > > > zzBuffer if
>> >> > > > > > > it
>> >> > > > > > > > > exceeded the threshold. We can decide on a reasonable
>> >> > > > > > > > > 64K
>> >> > > > threshold
>> >> > > > > > > or
>> >> > > > > > > > > something, or simply always cut back to 16 KB. As far
>> >> > > > > > > > > as
>> >> > I
>> >> > > > > > > understand,
>> >> > > > > > > > that
>> >> > > > > > > > > buffer should never grow that much. I.e. in zzRefill,
>> >> > which
>> >> > > > is the
>> >> > > > > > > only
>> >> > > > > > > > > place where the buffer gets resized, there is an
>> >> > > > > > > > > attempt
>> >> > to
>> >> > > > first
>> >> > > > > > > move
>> >> > > > > > > > back
>> >> > > > > > > > > characters that were already consumed and only then
>> >> > allocate
>> >> > > > a
>> >> > > > > > > bigger
>> >> > > > > > > > > buffer. Which means only if there is a token whose size
>> >> > is
>> >> > > > larger
>> >> > > > > > > than
>> >> > > > > > > > 16KB
>> >> > > > > > > > > (!?), will this buffer get expanded.
>> >> > > > > > > > >
>> >> > > > > > > > > A trimBuffer method might not be that bad .. as a
>> >> > protective
>> >> > > > > > > measure.
>> >> > > > > > > > What
>> >> > > > > > > > > do you think? Of course, JFlex can fix it on their own
>> >> > ...
>> >> > > > but
>> >> > > > > > > until that
>> >> > > > > > > > > happens ...
>> >> > > > > > > > >
>> >> > > > > > > > > Shai
>> >> > > > > > > > >
>> >> > > > > > > > > On Thu, Apr 8, 2010 at 10:35 AM, Uwe Schindler
>> >> > > > <uw...@thetaphi.de>
>> >> > > > > > > wrote:
>> >> > > > > > > > >
>> >> > > > > > > > > > > I would like to identify also the problematic
>> >> > document I
>> >> > > > have
>> >> > > > > > > 10000
>> >> > > > > > > > so,
>> >> > > > > > > > > > > what
>> >> > > > > > > > > > > would be the best way of identifying the one that
>> >> > > > > > > > > > > it
>> >> > > > making
>> >> > > > > > > zzBuffer
>> >> > > > > > > > to
>> >> > > > > > > > > > > grow
>> >> > > > > > > > > > > without control?
>> >> > > > > > > > > >
>> >> > > > > > > > > > Dont index your documents, but instead pass them
>> >> > directly
>> >> > > > to the
>> >> > > > > > > > analyzer
>> >> > > > > > > > > > and consume the tokenstream manually. Then visit
>> >> > > > > > > > > TermAttribute.termLength()
>> >> > > > > > > > > > for each Token.
>> >> > > > > > > > > >
>> >> > > > > > > > > >
>> >> > > > > > > > > >
>> >> > > > > > > > > > -------------------------------------------------------
>> >> > ----
>> >> > > > ------
>> >> > > > > > > ----
>> >> > > > > > > > > > To unsubscribe, e-mail: java-user-
>> >> > > > unsubscribe@lucene.apache.org
>> >> > > > > > > > > > For additional commands, e-mail:
>> >> > > > > java-user-help@lucene.apache.org
>> >> > > > > > > > > >
>> >> > > > > > > > > >
>> >> > > > > > > > >
>> >> > > > > > > >
>> >> > > > > > > >
>> >> > > > > > > >
>> >> > > > > > > > --
>> >> > > > > > > > /Rubén
>> >> > > > > > > >
>> >> > > > > >
>> >> > > > > >
>> >> > > > > >
>> >> > > > > > ---------------------------------------------------------------
>> >> > ----
>> >> > > > --
>> >> > > > > > To unsubscribe, e-mail:
>> >> > > > > > java-user-unsubscribe@lucene.apache.org
>> >> > > > > > For additional commands, e-mail: java-user-
>> >> > help@lucene.apache.org
>> >> > > > > >
>> >> > > > > >
>> >> > > > >
>> >> > > >
>> >> > > >
>> >> > > >
>> >> > > > --
>> >> > > > /Rubén
>> >> > >
>> >> > >
>> >> > >
>> >> > > ---------------------------------------------------------------------
>> >> > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> >> > > For additional commands, e-mail: java-user-help@lucene.apache.org
>> >> > >
>> >> > >
>> >> >
>> >> >
>> >> > --
>> >> > /Rubén
>> >>
>> >>
>> >> ---------------------------------------------------------------------
>> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> >> For additional commands, e-mail: java-user-help@lucene.apache.org
>> >>
>> >>
>> >
>> >
>> > --
>> > /Rubén
>> >
>
>
>
> --
> /Rubén
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: IndexWriter memory leak?
Posted by Michael McCandless <lu...@mikemccandless.com>.
I agree IW should not hold refs to the Field instances from the last
doc indexed... I put a patch on LUCENE-2387 to null the reference as
we go. Can you confirm this lets GC reclaim?
Mike
On Fri, Apr 9, 2010 at 12:54 AM, Ruben Laguna <ru...@gmail.com> wrote:
> But the Readers I'm talking about are not held by the Tokenizer (at least
> not *only* by it), these are held by the DocFieldProccessorPerThread....
>
> IndexWriter -> DocumentsWriter -> DocumentsWriterThreadState ->
> DocFieldProcessorPerThread -> DocFieldProcessorPerField -> Fieldable ->
> Field (fieldsData)
>
> and it's not only one Reader there are several one (one per thread I
> suppose, in my heapdump there is 25 Reader that should have been GCed
> otherwise).
>
> Best regards/Ruben
> On Thu, Apr 8, 2010 at 11:49 PM, Uwe Schindler <uw...@thetaphi.de> wrote:
>
>> There is one possibility, that could be fixed:
>>
>> As Tokenizers are reused, the analyzer holds a reference to the last used
>> Reader. The easy fix would be to unset the Reader in Tokenizer.close(). If
>> this is the case for you, that may be easy to do. So Tokenizer.close() looks
>> like this:
>>
>> /** By default, closes the input Reader. */
>> @Override
>> public void close() throws IOException {
>> input.close();
>> input = null; // <-- new!
>> }
>>
>> Any comments from other committers?
>>
>> -----
>> Uwe Schindler
>> H.-H.-Meier-Allee 63, D-28213 Bremen
>> http://www.thetaphi.de
>> eMail: uwe@thetaphi.de
>>
>>
>> > -----Original Message-----
>> > From: Ruben Laguna [mailto:ruben.laguna@gmail.com]
>> > Sent: Thursday, April 08, 2010 2:50 PM
>> > To: java-user@lucene.apache.org
>> > Subject: Re: IndexWriter memory leak?
>> >
>> > I will double check in the afternoon the heapdump.hprof. But I think
>> > that
>> > *some* readers are indeed held by
>> > docWriter.threadStates[0].consumer.fieldHash[1].fields[xxxx],
>> > as shown in [1] (this heapdump contains only live objects). The
>> > heapdump
>> > was taken after IndexWriter.commit() /IndexWriter.optimize() and all
>> > the
>> > Documents were already indexed and GCed (I will double check).
>> >
>> > So that would mean that the Reader is retained in memory by the
>> > following
>> > chaing of references,
>> >
>> > DocumentsWriter -> DocumentsWriterThreadState ->
>> > DocFieldProcessorPerThread
>> > -> DocFieldProcessorPerField -> Fieldable -> Field (fieldsData)
>> >
>> > I'll double check with Eclipse MAT as I said that this chain is
>> > actually
>> > made of hard references only (no SoftReferences,WeakReferences, etc). I
>> > will
>> > also double check also that there is no "live" Document that is
>> > referencing
>> > the Reader via the Field.
>> >
>> >
>> > [1] http://img.skitch.com/20100407-b86irkp7e4uif2wq1dd4t899qb.jpg
>> >
>> > On Thu, Apr 8, 2010 at 2:16 PM, Uwe Schindler <uw...@thetaphi.de> wrote:
>> >
>> > > Readers are not held. If you indexed the document and gced the
>> > document
>> > > instance they readers are gone.
>> > >
>> > > -----
>> > > Uwe Schindler
>> > > H.-H.-Meier-Allee 63, D-28213 Bremen
>> > > http://www.thetaphi.de
>> > > eMail: uwe@thetaphi.de
>> > >
>> > >
>> > > > -----Original Message-----
>> > > > From: Ruben Laguna [mailto:ruben.laguna@gmail.com]
>> > > > Sent: Thursday, April 08, 2010 1:28 PM
>> > > > To: java-user@lucene.apache.org
>> > > > Subject: Re: IndexWriter memory leak?
>> > > >
>> > > > Now that the zzBuffer issue is solved...
>> > > >
>> > > > what about the references to the Readers held by docWriter. Tika´s
>> > > > ParsingReaders are quite heavyweight so retaining those in memory
>> > > > unnecesarily is also a "hidden" memory leak. Should I open a bug
>> > report
>> > > > on
>> > > > that one?
>> > > >
>> > > > /Rubén
>> > > >
>> > > > On Thu, Apr 8, 2010 at 12:11 PM, Shai Erera <se...@gmail.com>
>> > wrote:
>> > > >
>> > > > > Guess we were replying at the same time :).
>> > > > >
>> > > > > On Thu, Apr 8, 2010 at 1:04 PM, Uwe Schindler <uw...@thetaphi.de>
>> > > > wrote:
>> > > > >
>> > > > > > I already answered, that I will take care of this!
>> > > > > >
>> > > > > > Uwe
>> > > > > >
>> > > > > > -----
>> > > > > > Uwe Schindler
>> > > > > > H.-H.-Meier-Allee 63, D-28213 Bremen
>> > > > > > http://www.thetaphi.de
>> > > > > > eMail: uwe@thetaphi.de
>> > > > > >
>> > > > > >
>> > > > > > > -----Original Message-----
>> > > > > > > From: Shai Erera [mailto:serera@gmail.com]
>> > > > > > > Sent: Thursday, April 08, 2010 12:00 PM
>> > > > > > > To: java-user@lucene.apache.org
>> > > > > > > Subject: Re: IndexWriter memory leak?
>> > > > > > >
>> > > > > > > Yes, that's the trimBuffer version I was thinking about, only
>> > > > this guy
>> > > > > > > created a reset(Reader, int) and does both ops (resetting +
>> > trim)
>> > > > in
>> > > > > > > one
>> > > > > > > method call. More convenient. Can you please open an issue to
>> > > > track
>> > > > > > > that?
>> > > > > > > People will have a chance to comment on whether we (Lucene)
>> > > > should
>> > > > > > > handle
>> > > > > > > that, or it should be a JFlex fix. Based on the number of
>> > replies
>> > > > this
>> > > > > > > guy
>> > > > > > > received (0 !), I doubt JFlex would consider it a problem.
>> > But we
>> > > > can
>> > > > > > > do
>> > > > > > > some small service to our users base by protecting against
>> > such
>> > > > > > > problems.
>> > > > > > >
>> > > > > > > And while you're opening the issue, if you want to take a
>> > stab at
>> > > > > > > fixing it
>> > > > > > > and post a patch, it'd be great :).
>> > > > > > >
>> > > > > > > Shai
>> > > > > > >
>> > > > > > > On Thu, Apr 8, 2010 at 12:51 PM, Ruben Laguna
>> > > > > > > <ru...@gmail.com>wrote:
>> > > > > > >
>> > > > > > > > I was investigating this a little further and in the JFlex
>> > > > mailing
>> > > > > > > list I
>> > > > > > > > found [1]
>> > > > > > > >
>> > > > > > > > I don't know much about flex / JFlex but it seems that this
>> > guy
>> > > > > > > resets the
>> > > > > > > > zzBuffer to 16384 or less when setting the input for the
>> > lexer
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > Quoted from shef <sh...@ya...>
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > I set
>> > > > > > > >
>> > > > > > > > %buffer 0
>> > > > > > > >
>> > > > > > > > in the options section, and then added this method to the
>> > > > lexer:
>> > > > > > > >
>> > > > > > > > /**
>> > > > > > > > * Set the input for the lexer. The size parameter
>> > really
>> > > > speeds
>> > > > > > > things
>> > > > > > > > up,
>> > > > > > > > * because by default, the lexer allocates an internal
>> > > > buffer of
>> > > > > > > 16k.
>> > > > > > > > For
>> > > > > > > > * most strings, this is unnecessarily large. If the
>> > size
>> > > > param is
>> > > > > > > > 0 or greater
>> > > > > > > > * than 16k, then the buffer is set to 16k. If the size
>> > > > param is
>> > > > > > > > smaller, then
>> > > > > > > > * the buf will be set to the exact size.
>> > > > > > > > * @param r the reader that provides the data
>> > > > > > > > * @param the size of the data in the reader.
>> > > > > > > > */
>> > > > > > > > public void reset(Reader r, int size) {
>> > > > > > > > if (size == 0 || size > 16384)
>> > > > > > > > size = 16384;
>> > > > > > > > zzBuffer = new char[size];
>> > > > > > > > yyreset(r);
>> > > > > > > > }
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > So maybe there is a way to trim the zzBuffer this way (?).
>> > > > > > > >
>> > > > > > > > BTW, I will try to find out which is the "big token" in my
>> > > > dataset
>> > > > > > > this
>> > > > > > > > afternoon. Thanks for the help.
>> > > > > > > >
>> > > > > > > > I actually workaround this memory problem for the time
>> > being by
>> > > > > > > wrapping
>> > > > > > > > the
>> > > > > > > > IndexWriter in a class that periodically closes the
>> > IndexWriter
>> > > > and
>> > > > > > > creates
>> > > > > > > > a new one, allowing the old to be GCed, but I would be
>> > really
>> > > > good if
>> > > > > > > > either
>> > > > > > > > JFlex or Lucene can take care of this zzBuffer going
>> > berserk.
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > Again thanks for the quick response. /Rubén
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > [1]
>> > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > >
>> > > >
>> > https://sourceforge.net/mailarchive/message.php?msg_id=444070.38422.qm@
>> > > > > > > web38901.mail.mud.yahoo.com
>> > > > > > > >
>> > > > > > > > On Thu, Apr 8, 2010 at 11:32 AM, Shai Erera
>> > <se...@gmail.com>
>> > > > > wrote:
>> > > > > > > >
>> > > > > > > > > If we could change the Flex file so that yyreset(Reader)
>> > > > would
>> > > > > > > check the
>> > > > > > > > > size of zzBuffer, we could trim it when it gets too big.
>> > But
>> > > > I
>> > > > > > > don't
>> > > > > > > > think
>> > > > > > > > > we have such control when writing the flex syntax ...
>> > yyreset
>> > > > is
>> > > > > > > > generated
>> > > > > > > > > by JFlex and that's the only place I can think of to trim
>> > the
>> > > > > > > buffer down
>> > > > > > > > > when it exceeds a predefined threshold ....
>> > > > > > > > >
>> > > > > > > > > Maybe what we can do is create our own method which will
>> > be
>> > > > called
>> > > > > > > by
>> > > > > > > > > StandardTokenizer after yyreset is called, something like
>> > > > > > > > > trimBufferIfTooBig(int threshold) which will reallocate
>> > > > zzBuffer if
>> > > > > > > it
>> > > > > > > > > exceeded the threshold. We can decide on a reasonable 64K
>> > > > threshold
>> > > > > > > or
>> > > > > > > > > something, or simply always cut back to 16 KB. As far as
>> > I
>> > > > > > > understand,
>> > > > > > > > that
>> > > > > > > > > buffer should never grow that much. I.e. in zzRefill,
>> > which
>> > > > is the
>> > > > > > > only
>> > > > > > > > > place where the buffer gets resized, there is an attempt
>> > to
>> > > > first
>> > > > > > > move
>> > > > > > > > back
>> > > > > > > > > characters that were already consumed and only then
>> > allocate
>> > > > a
>> > > > > > > bigger
>> > > > > > > > > buffer. Which means only if there is a token whose size
>> > is
>> > > > larger
>> > > > > > > than
>> > > > > > > > 16KB
>> > > > > > > > > (!?), will this buffer get expanded.
>> > > > > > > > >
>> > > > > > > > > A trimBuffer method might not be that bad .. as a
>> > protective
>> > > > > > > measure.
>> > > > > > > > What
>> > > > > > > > > do you think? Of course, JFlex can fix it on their own
>> > ...
>> > > > but
>> > > > > > > until that
>> > > > > > > > > happens ...
>> > > > > > > > >
>> > > > > > > > > Shai
>> > > > > > > > >
>> > > > > > > > > On Thu, Apr 8, 2010 at 10:35 AM, Uwe Schindler
>> > > > <uw...@thetaphi.de>
>> > > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > > > > I would like to identify also the problematic
>> > document I
>> > > > have
>> > > > > > > 10000
>> > > > > > > > so,
>> > > > > > > > > > > what
>> > > > > > > > > > > would be the best way of identifying the one that it
>> > > > making
>> > > > > > > zzBuffer
>> > > > > > > > to
>> > > > > > > > > > > grow
>> > > > > > > > > > > without control?
>> > > > > > > > > >
>> > > > > > > > > > Dont index your documents, but instead pass them
>> > directly
>> > > > to the
>> > > > > > > > analyzer
>> > > > > > > > > > and consume the tokenstream manually. Then visit
>> > > > > > > > > TermAttribute.termLength()
>> > > > > > > > > > for each Token.
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > -------------------------------------------------------
>> > ----
>> > > > ------
>> > > > > > > ----
>> > > > > > > > > > To unsubscribe, e-mail: java-user-
>> > > > unsubscribe@lucene.apache.org
>> > > > > > > > > > For additional commands, e-mail:
>> > > > > java-user-help@lucene.apache.org
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > --
>> > > > > > > > /Rubén
>> > > > > > > >
>> > > > > >
>> > > > > >
>> > > > > > ---------------------------------------------------------------
>> > ----
>> > > > --
>> > > > > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> > > > > > For additional commands, e-mail: java-user-
>> > help@lucene.apache.org
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > > >
>> > > >
>> > > > --
>> > > > /Rubén
>> > >
>> > >
>> > > ---------------------------------------------------------------------
>> > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> > > For additional commands, e-mail: java-user-help@lucene.apache.org
>> > >
>> > >
>> >
>> >
>> > --
>> > /Rubén
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
>
> --
> /Rubén
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: IndexWriter memory leak?
Posted by Ruben Laguna <ru...@gmail.com>.
But the Readers I'm talking about are not held by the Tokenizer (at least
not *only* by it), these are held by the DocFieldProccessorPerThread....
IndexWriter -> DocumentsWriter -> DocumentsWriterThreadState ->
DocFieldProcessorPerThread -> DocFieldProcessorPerField -> Fieldable ->
Field (fieldsData)
and it's not only one Reader there are several one (one per thread I
suppose, in my heapdump there is 25 Reader that should have been GCed
otherwise).
Best regards/Ruben
On Thu, Apr 8, 2010 at 11:49 PM, Uwe Schindler <uw...@thetaphi.de> wrote:
> There is one possibility, that could be fixed:
>
> As Tokenizers are reused, the analyzer holds a reference to the last used
> Reader. The easy fix would be to unset the Reader in Tokenizer.close(). If
> this is the case for you, that may be easy to do. So Tokenizer.close() looks
> like this:
>
> /** By default, closes the input Reader. */
> @Override
> public void close() throws IOException {
> input.close();
> input = null; // <-- new!
> }
>
> Any comments from other committers?
>
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: uwe@thetaphi.de
>
>
> > -----Original Message-----
> > From: Ruben Laguna [mailto:ruben.laguna@gmail.com]
> > Sent: Thursday, April 08, 2010 2:50 PM
> > To: java-user@lucene.apache.org
> > Subject: Re: IndexWriter memory leak?
> >
> > I will double check in the afternoon the heapdump.hprof. But I think
> > that
> > *some* readers are indeed held by
> > docWriter.threadStates[0].consumer.fieldHash[1].fields[xxxx],
> > as shown in [1] (this heapdump contains only live objects). The
> > heapdump
> > was taken after IndexWriter.commit() /IndexWriter.optimize() and all
> > the
> > Documents were already indexed and GCed (I will double check).
> >
> > So that would mean that the Reader is retained in memory by the
> > following
> > chaing of references,
> >
> > DocumentsWriter -> DocumentsWriterThreadState ->
> > DocFieldProcessorPerThread
> > -> DocFieldProcessorPerField -> Fieldable -> Field (fieldsData)
> >
> > I'll double check with Eclipse MAT as I said that this chain is
> > actually
> > made of hard references only (no SoftReferences,WeakReferences, etc). I
> > will
> > also double check also that there is no "live" Document that is
> > referencing
> > the Reader via the Field.
> >
> >
> > [1] http://img.skitch.com/20100407-b86irkp7e4uif2wq1dd4t899qb.jpg
> >
> > On Thu, Apr 8, 2010 at 2:16 PM, Uwe Schindler <uw...@thetaphi.de> wrote:
> >
> > > Readers are not held. If you indexed the document and gced the
> > document
> > > instance they readers are gone.
> > >
> > > -----
> > > Uwe Schindler
> > > H.-H.-Meier-Allee 63, D-28213 Bremen
> > > http://www.thetaphi.de
> > > eMail: uwe@thetaphi.de
> > >
> > >
> > > > -----Original Message-----
> > > > From: Ruben Laguna [mailto:ruben.laguna@gmail.com]
> > > > Sent: Thursday, April 08, 2010 1:28 PM
> > > > To: java-user@lucene.apache.org
> > > > Subject: Re: IndexWriter memory leak?
> > > >
> > > > Now that the zzBuffer issue is solved...
> > > >
> > > > what about the references to the Readers held by docWriter. Tika´s
> > > > ParsingReaders are quite heavyweight so retaining those in memory
> > > > unnecesarily is also a "hidden" memory leak. Should I open a bug
> > report
> > > > on
> > > > that one?
> > > >
> > > > /Rubén
> > > >
> > > > On Thu, Apr 8, 2010 at 12:11 PM, Shai Erera <se...@gmail.com>
> > wrote:
> > > >
> > > > > Guess we were replying at the same time :).
> > > > >
> > > > > On Thu, Apr 8, 2010 at 1:04 PM, Uwe Schindler <uw...@thetaphi.de>
> > > > wrote:
> > > > >
> > > > > > I already answered, that I will take care of this!
> > > > > >
> > > > > > Uwe
> > > > > >
> > > > > > -----
> > > > > > Uwe Schindler
> > > > > > H.-H.-Meier-Allee 63, D-28213 Bremen
> > > > > > http://www.thetaphi.de
> > > > > > eMail: uwe@thetaphi.de
> > > > > >
> > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: Shai Erera [mailto:serera@gmail.com]
> > > > > > > Sent: Thursday, April 08, 2010 12:00 PM
> > > > > > > To: java-user@lucene.apache.org
> > > > > > > Subject: Re: IndexWriter memory leak?
> > > > > > >
> > > > > > > Yes, that's the trimBuffer version I was thinking about, only
> > > > this guy
> > > > > > > created a reset(Reader, int) and does both ops (resetting +
> > trim)
> > > > in
> > > > > > > one
> > > > > > > method call. More convenient. Can you please open an issue to
> > > > track
> > > > > > > that?
> > > > > > > People will have a chance to comment on whether we (Lucene)
> > > > should
> > > > > > > handle
> > > > > > > that, or it should be a JFlex fix. Based on the number of
> > replies
> > > > this
> > > > > > > guy
> > > > > > > received (0 !), I doubt JFlex would consider it a problem.
> > But we
> > > > can
> > > > > > > do
> > > > > > > some small service to our users base by protecting against
> > such
> > > > > > > problems.
> > > > > > >
> > > > > > > And while you're opening the issue, if you want to take a
> > stab at
> > > > > > > fixing it
> > > > > > > and post a patch, it'd be great :).
> > > > > > >
> > > > > > > Shai
> > > > > > >
> > > > > > > On Thu, Apr 8, 2010 at 12:51 PM, Ruben Laguna
> > > > > > > <ru...@gmail.com>wrote:
> > > > > > >
> > > > > > > > I was investigating this a little further and in the JFlex
> > > > mailing
> > > > > > > list I
> > > > > > > > found [1]
> > > > > > > >
> > > > > > > > I don't know much about flex / JFlex but it seems that this
> > guy
> > > > > > > resets the
> > > > > > > > zzBuffer to 16384 or less when setting the input for the
> > lexer
> > > > > > > >
> > > > > > > >
> > > > > > > > Quoted from shef <sh...@ya...>
> > > > > > > >
> > > > > > > >
> > > > > > > > I set
> > > > > > > >
> > > > > > > > %buffer 0
> > > > > > > >
> > > > > > > > in the options section, and then added this method to the
> > > > lexer:
> > > > > > > >
> > > > > > > > /**
> > > > > > > > * Set the input for the lexer. The size parameter
> > really
> > > > speeds
> > > > > > > things
> > > > > > > > up,
> > > > > > > > * because by default, the lexer allocates an internal
> > > > buffer of
> > > > > > > 16k.
> > > > > > > > For
> > > > > > > > * most strings, this is unnecessarily large. If the
> > size
> > > > param is
> > > > > > > > 0 or greater
> > > > > > > > * than 16k, then the buffer is set to 16k. If the size
> > > > param is
> > > > > > > > smaller, then
> > > > > > > > * the buf will be set to the exact size.
> > > > > > > > * @param r the reader that provides the data
> > > > > > > > * @param the size of the data in the reader.
> > > > > > > > */
> > > > > > > > public void reset(Reader r, int size) {
> > > > > > > > if (size == 0 || size > 16384)
> > > > > > > > size = 16384;
> > > > > > > > zzBuffer = new char[size];
> > > > > > > > yyreset(r);
> > > > > > > > }
> > > > > > > >
> > > > > > > >
> > > > > > > > So maybe there is a way to trim the zzBuffer this way (?).
> > > > > > > >
> > > > > > > > BTW, I will try to find out which is the "big token" in my
> > > > dataset
> > > > > > > this
> > > > > > > > afternoon. Thanks for the help.
> > > > > > > >
> > > > > > > > I actually workaround this memory problem for the time
> > being by
> > > > > > > wrapping
> > > > > > > > the
> > > > > > > > IndexWriter in a class that periodically closes the
> > IndexWriter
> > > > and
> > > > > > > creates
> > > > > > > > a new one, allowing the old to be GCed, but I would be
> > really
> > > > good if
> > > > > > > > either
> > > > > > > > JFlex or Lucene can take care of this zzBuffer going
> > berserk.
> > > > > > > >
> > > > > > > >
> > > > > > > > Again thanks for the quick response. /Rubén
> > > > > > > >
> > > > > > > >
> > > > > > > > [1]
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > > >
> > https://sourceforge.net/mailarchive/message.php?msg_id=444070.38422.qm@
> > > > > > > web38901.mail.mud.yahoo.com
> > > > > > > >
> > > > > > > > On Thu, Apr 8, 2010 at 11:32 AM, Shai Erera
> > <se...@gmail.com>
> > > > > wrote:
> > > > > > > >
> > > > > > > > > If we could change the Flex file so that yyreset(Reader)
> > > > would
> > > > > > > check the
> > > > > > > > > size of zzBuffer, we could trim it when it gets too big.
> > But
> > > > I
> > > > > > > don't
> > > > > > > > think
> > > > > > > > > we have such control when writing the flex syntax ...
> > yyreset
> > > > is
> > > > > > > > generated
> > > > > > > > > by JFlex and that's the only place I can think of to trim
> > the
> > > > > > > buffer down
> > > > > > > > > when it exceeds a predefined threshold ....
> > > > > > > > >
> > > > > > > > > Maybe what we can do is create our own method which will
> > be
> > > > called
> > > > > > > by
> > > > > > > > > StandardTokenizer after yyreset is called, something like
> > > > > > > > > trimBufferIfTooBig(int threshold) which will reallocate
> > > > zzBuffer if
> > > > > > > it
> > > > > > > > > exceeded the threshold. We can decide on a reasonable 64K
> > > > threshold
> > > > > > > or
> > > > > > > > > something, or simply always cut back to 16 KB. As far as
> > I
> > > > > > > understand,
> > > > > > > > that
> > > > > > > > > buffer should never grow that much. I.e. in zzRefill,
> > which
> > > > is the
> > > > > > > only
> > > > > > > > > place where the buffer gets resized, there is an attempt
> > to
> > > > first
> > > > > > > move
> > > > > > > > back
> > > > > > > > > characters that were already consumed and only then
> > allocate
> > > > a
> > > > > > > bigger
> > > > > > > > > buffer. Which means only if there is a token whose size
> > is
> > > > larger
> > > > > > > than
> > > > > > > > 16KB
> > > > > > > > > (!?), will this buffer get expanded.
> > > > > > > > >
> > > > > > > > > A trimBuffer method might not be that bad .. as a
> > protective
> > > > > > > measure.
> > > > > > > > What
> > > > > > > > > do you think? Of course, JFlex can fix it on their own
> > ...
> > > > but
> > > > > > > until that
> > > > > > > > > happens ...
> > > > > > > > >
> > > > > > > > > Shai
> > > > > > > > >
> > > > > > > > > On Thu, Apr 8, 2010 at 10:35 AM, Uwe Schindler
> > > > <uw...@thetaphi.de>
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > > I would like to identify also the problematic
> > document I
> > > > have
> > > > > > > 10000
> > > > > > > > so,
> > > > > > > > > > > what
> > > > > > > > > > > would be the best way of identifying the one that it
> > > > making
> > > > > > > zzBuffer
> > > > > > > > to
> > > > > > > > > > > grow
> > > > > > > > > > > without control?
> > > > > > > > > >
> > > > > > > > > > Dont index your documents, but instead pass them
> > directly
> > > > to the
> > > > > > > > analyzer
> > > > > > > > > > and consume the tokenstream manually. Then visit
> > > > > > > > > TermAttribute.termLength()
> > > > > > > > > > for each Token.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > -------------------------------------------------------
> > ----
> > > > ------
> > > > > > > ----
> > > > > > > > > > To unsubscribe, e-mail: java-user-
> > > > unsubscribe@lucene.apache.org
> > > > > > > > > > For additional commands, e-mail:
> > > > > java-user-help@lucene.apache.org
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > /Rubén
> > > > > > > >
> > > > > >
> > > > > >
> > > > > > ---------------------------------------------------------------
> > ----
> > > > --
> > > > > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > > > > > For additional commands, e-mail: java-user-
> > help@lucene.apache.org
> > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > /Rubén
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > > For additional commands, e-mail: java-user-help@lucene.apache.org
> > >
> > >
> >
> >
> > --
> > /Rubén
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
--
/Rubén
Re: IndexWriter memory leak?
Posted by Ruben Laguna <ru...@gmail.com>.
Yeah, I checked again and IndexWriter is holding references to the Reader,
I'm afraid.
I opened bug report https://issues.apache.org/jira/browse/LUCENE-2387 to
track this down.
On Thu, Apr 8, 2010 at 2:50 PM, Ruben Laguna <ru...@gmail.com> wrote:
> I will double check in the afternoon the heapdump.hprof. But I think that
> *some* readers are indeed held by docWriter.threadStates[0].consumer.fieldHash[1].fields[xxxx],
> as shown in [1] (this heapdump contains only live objects). The heapdump
> was taken after IndexWriter.commit() /IndexWriter.optimize() and all the
> Documents were already indexed and GCed (I will double check).
>
> So that would mean that the Reader is retained in memory by the following
> chaing of references,
>
> DocumentsWriter -> DocumentsWriterThreadState -> DocFieldProcessorPerThread
> -> DocFieldProcessorPerField -> Fieldable -> Field (fieldsData)
>
> I'll double check with Eclipse MAT as I said that this chain is actually
> made of hard references only (no SoftReferences,WeakReferences, etc). I will
> also double check also that there is no "live" Document that is referencing
> the Reader via the Field.
>
>
> [1] http://img.skitch.com/20100407-b86irkp7e4uif2wq1dd4t899qb.jpg
>
> On Thu, Apr 8, 2010 at 2:16 PM, Uwe Schindler <uw...@thetaphi.de> wrote:
>
>> Readers are not held. If you indexed the document and gced the document
>> instance they readers are gone.
>>
>> -----
>> Uwe Schindler
>> H.-H.-Meier-Allee 63, D-28213 Bremen
>> http://www.thetaphi.de
>> eMail: uwe@thetaphi.de
>>
>>
>> > -----Original Message-----
>> > From: Ruben Laguna [mailto:ruben.laguna@gmail.com]
>> > Sent: Thursday, April 08, 2010 1:28 PM
>> > To: java-user@lucene.apache.org
>> > Subject: Re: IndexWriter memory leak?
>> >
>> > Now that the zzBuffer issue is solved...
>> >
>> > what about the references to the Readers held by docWriter. Tika´s
>> > ParsingReaders are quite heavyweight so retaining those in memory
>> > unnecesarily is also a "hidden" memory leak. Should I open a bug report
>> > on
>> > that one?
>> >
>> > /Rubén
>> >
>> > On Thu, Apr 8, 2010 at 12:11 PM, Shai Erera <se...@gmail.com> wrote:
>> >
>> > > Guess we were replying at the same time :).
>> > >
>> > > On Thu, Apr 8, 2010 at 1:04 PM, Uwe Schindler <uw...@thetaphi.de>
>> > wrote:
>> > >
>> > > > I already answered, that I will take care of this!
>> > > >
>> > > > Uwe
>> > > >
>> > > > -----
>> > > > Uwe Schindler
>> > > > H.-H.-Meier-Allee 63, D-28213 Bremen
>> > > > http://www.thetaphi.de
>> > > > eMail: uwe@thetaphi.de
>> > > >
>> > > >
>> > > > > -----Original Message-----
>> > > > > From: Shai Erera [mailto:serera@gmail.com]
>> > > > > Sent: Thursday, April 08, 2010 12:00 PM
>> > > > > To: java-user@lucene.apache.org
>> > > > > Subject: Re: IndexWriter memory leak?
>> > > > >
>> > > > > Yes, that's the trimBuffer version I was thinking about, only
>> > this guy
>> > > > > created a reset(Reader, int) and does both ops (resetting + trim)
>> > in
>> > > > > one
>> > > > > method call. More convenient. Can you please open an issue to
>> > track
>> > > > > that?
>> > > > > People will have a chance to comment on whether we (Lucene)
>> > should
>> > > > > handle
>> > > > > that, or it should be a JFlex fix. Based on the number of replies
>> > this
>> > > > > guy
>> > > > > received (0 !), I doubt JFlex would consider it a problem. But we
>> > can
>> > > > > do
>> > > > > some small service to our users base by protecting against such
>> > > > > problems.
>> > > > >
>> > > > > And while you're opening the issue, if you want to take a stab at
>> > > > > fixing it
>> > > > > and post a patch, it'd be great :).
>> > > > >
>> > > > > Shai
>> > > > >
>> > > > > On Thu, Apr 8, 2010 at 12:51 PM, Ruben Laguna
>> > > > > <ru...@gmail.com>wrote:
>> > > > >
>> > > > > > I was investigating this a little further and in the JFlex
>> > mailing
>> > > > > list I
>> > > > > > found [1]
>> > > > > >
>> > > > > > I don't know much about flex / JFlex but it seems that this guy
>> > > > > resets the
>> > > > > > zzBuffer to 16384 or less when setting the input for the lexer
>> > > > > >
>> > > > > >
>> > > > > > Quoted from shef <sh...@ya...>
>> > > > > >
>> > > > > >
>> > > > > > I set
>> > > > > >
>> > > > > > %buffer 0
>> > > > > >
>> > > > > > in the options section, and then added this method to the
>> > lexer:
>> > > > > >
>> > > > > > /**
>> > > > > > * Set the input for the lexer. The size parameter really
>> > speeds
>> > > > > things
>> > > > > > up,
>> > > > > > * because by default, the lexer allocates an internal
>> > buffer of
>> > > > > 16k.
>> > > > > > For
>> > > > > > * most strings, this is unnecessarily large. If the size
>> > param is
>> > > > > > 0 or greater
>> > > > > > * than 16k, then the buffer is set to 16k. If the size
>> > param is
>> > > > > > smaller, then
>> > > > > > * the buf will be set to the exact size.
>> > > > > > * @param r the reader that provides the data
>> > > > > > * @param the size of the data in the reader.
>> > > > > > */
>> > > > > > public void reset(Reader r, int size) {
>> > > > > > if (size == 0 || size > 16384)
>> > > > > > size = 16384;
>> > > > > > zzBuffer = new char[size];
>> > > > > > yyreset(r);
>> > > > > > }
>> > > > > >
>> > > > > >
>> > > > > > So maybe there is a way to trim the zzBuffer this way (?).
>> > > > > >
>> > > > > > BTW, I will try to find out which is the "big token" in my
>> > dataset
>> > > > > this
>> > > > > > afternoon. Thanks for the help.
>> > > > > >
>> > > > > > I actually workaround this memory problem for the time being by
>> > > > > wrapping
>> > > > > > the
>> > > > > > IndexWriter in a class that periodically closes the IndexWriter
>> > and
>> > > > > creates
>> > > > > > a new one, allowing the old to be GCed, but I would be really
>> > good if
>> > > > > > either
>> > > > > > JFlex or Lucene can take care of this zzBuffer going berserk.
>> > > > > >
>> > > > > >
>> > > > > > Again thanks for the quick response. /Rubén
>> > > > > >
>> > > > > >
>> > > > > > [1]
>> > > > > >
>> > > > > >
>> > > > >
>> > >
>> > https://sourceforge.net/mailarchive/message.php?msg_id=444070.38422.qm@
>> > > > > web38901.mail.mud.yahoo.com
>> > > > > >
>> > > > > > On Thu, Apr 8, 2010 at 11:32 AM, Shai Erera <se...@gmail.com>
>> > > wrote:
>> > > > > >
>> > > > > > > If we could change the Flex file so that yyreset(Reader)
>> > would
>> > > > > check the
>> > > > > > > size of zzBuffer, we could trim it when it gets too big. But
>> > I
>> > > > > don't
>> > > > > > think
>> > > > > > > we have such control when writing the flex syntax ... yyreset
>> > is
>> > > > > > generated
>> > > > > > > by JFlex and that's the only place I can think of to trim the
>> > > > > buffer down
>> > > > > > > when it exceeds a predefined threshold ....
>> > > > > > >
>> > > > > > > Maybe what we can do is create our own method which will be
>> > called
>> > > > > by
>> > > > > > > StandardTokenizer after yyreset is called, something like
>> > > > > > > trimBufferIfTooBig(int threshold) which will reallocate
>> > zzBuffer if
>> > > > > it
>> > > > > > > exceeded the threshold. We can decide on a reasonable 64K
>> > threshold
>> > > > > or
>> > > > > > > something, or simply always cut back to 16 KB. As far as I
>> > > > > understand,
>> > > > > > that
>> > > > > > > buffer should never grow that much. I.e. in zzRefill, which
>> > is the
>> > > > > only
>> > > > > > > place where the buffer gets resized, there is an attempt to
>> > first
>> > > > > move
>> > > > > > back
>> > > > > > > characters that were already consumed and only then allocate
>> > a
>> > > > > bigger
>> > > > > > > buffer. Which means only if there is a token whose size is
>> > larger
>> > > > > than
>> > > > > > 16KB
>> > > > > > > (!?), will this buffer get expanded.
>> > > > > > >
>> > > > > > > A trimBuffer method might not be that bad .. as a protective
>> > > > > measure.
>> > > > > > What
>> > > > > > > do you think? Of course, JFlex can fix it on their own ...
>> > but
>> > > > > until that
>> > > > > > > happens ...
>> > > > > > >
>> > > > > > > Shai
>> > > > > > >
>> > > > > > > On Thu, Apr 8, 2010 at 10:35 AM, Uwe Schindler
>> > <uw...@thetaphi.de>
>> > > > > wrote:
>> > > > > > >
>> > > > > > > > > I would like to identify also the problematic document I
>> > have
>> > > > > 10000
>> > > > > > so,
>> > > > > > > > > what
>> > > > > > > > > would be the best way of identifying the one that it
>> > making
>> > > > > zzBuffer
>> > > > > > to
>> > > > > > > > > grow
>> > > > > > > > > without control?
>> > > > > > > >
>> > > > > > > > Dont index your documents, but instead pass them directly
>> > to the
>> > > > > > analyzer
>> > > > > > > > and consume the tokenstream manually. Then visit
>> > > > > > > TermAttribute.termLength()
>> > > > > > > > for each Token.
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > -----------------------------------------------------------
>> > ------
>> > > > > ----
>> > > > > > > > To unsubscribe, e-mail: java-user-
>> > unsubscribe@lucene.apache.org
>> > > > > > > > For additional commands, e-mail:
>> > > java-user-help@lucene.apache.org
>> > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > --
>> > > > > > /Rubén
>> > > > > >
>> > > >
>> > > >
>> > > > -------------------------------------------------------------------
>> > --
>> > > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> > > > For additional commands, e-mail: java-user-help@lucene.apache.org
>> > > >
>> > > >
>> > >
>> >
>> >
>> >
>> > --
>> > /Rubén
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
>
> --
> /Rubén
>
--
/Rubén
RE: IndexWriter memory leak?
Posted by Uwe Schindler <uw...@thetaphi.de>.
There is one possibility, that could be fixed:
As Tokenizers are reused, the analyzer holds a reference to the last used Reader. The easy fix would be to unset the Reader in Tokenizer.close(). If this is the case for you, that may be easy to do. So Tokenizer.close() looks like this:
/** By default, closes the input Reader. */
@Override
public void close() throws IOException {
input.close();
input = null; // <-- new!
}
Any comments from other committers?
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: Ruben Laguna [mailto:ruben.laguna@gmail.com]
> Sent: Thursday, April 08, 2010 2:50 PM
> To: java-user@lucene.apache.org
> Subject: Re: IndexWriter memory leak?
>
> I will double check in the afternoon the heapdump.hprof. But I think
> that
> *some* readers are indeed held by
> docWriter.threadStates[0].consumer.fieldHash[1].fields[xxxx],
> as shown in [1] (this heapdump contains only live objects). The
> heapdump
> was taken after IndexWriter.commit() /IndexWriter.optimize() and all
> the
> Documents were already indexed and GCed (I will double check).
>
> So that would mean that the Reader is retained in memory by the
> following
> chaing of references,
>
> DocumentsWriter -> DocumentsWriterThreadState ->
> DocFieldProcessorPerThread
> -> DocFieldProcessorPerField -> Fieldable -> Field (fieldsData)
>
> I'll double check with Eclipse MAT as I said that this chain is
> actually
> made of hard references only (no SoftReferences,WeakReferences, etc). I
> will
> also double check also that there is no "live" Document that is
> referencing
> the Reader via the Field.
>
>
> [1] http://img.skitch.com/20100407-b86irkp7e4uif2wq1dd4t899qb.jpg
>
> On Thu, Apr 8, 2010 at 2:16 PM, Uwe Schindler <uw...@thetaphi.de> wrote:
>
> > Readers are not held. If you indexed the document and gced the
> document
> > instance they readers are gone.
> >
> > -----
> > Uwe Schindler
> > H.-H.-Meier-Allee 63, D-28213 Bremen
> > http://www.thetaphi.de
> > eMail: uwe@thetaphi.de
> >
> >
> > > -----Original Message-----
> > > From: Ruben Laguna [mailto:ruben.laguna@gmail.com]
> > > Sent: Thursday, April 08, 2010 1:28 PM
> > > To: java-user@lucene.apache.org
> > > Subject: Re: IndexWriter memory leak?
> > >
> > > Now that the zzBuffer issue is solved...
> > >
> > > what about the references to the Readers held by docWriter. Tika´s
> > > ParsingReaders are quite heavyweight so retaining those in memory
> > > unnecesarily is also a "hidden" memory leak. Should I open a bug
> report
> > > on
> > > that one?
> > >
> > > /Rubén
> > >
> > > On Thu, Apr 8, 2010 at 12:11 PM, Shai Erera <se...@gmail.com>
> wrote:
> > >
> > > > Guess we were replying at the same time :).
> > > >
> > > > On Thu, Apr 8, 2010 at 1:04 PM, Uwe Schindler <uw...@thetaphi.de>
> > > wrote:
> > > >
> > > > > I already answered, that I will take care of this!
> > > > >
> > > > > Uwe
> > > > >
> > > > > -----
> > > > > Uwe Schindler
> > > > > H.-H.-Meier-Allee 63, D-28213 Bremen
> > > > > http://www.thetaphi.de
> > > > > eMail: uwe@thetaphi.de
> > > > >
> > > > >
> > > > > > -----Original Message-----
> > > > > > From: Shai Erera [mailto:serera@gmail.com]
> > > > > > Sent: Thursday, April 08, 2010 12:00 PM
> > > > > > To: java-user@lucene.apache.org
> > > > > > Subject: Re: IndexWriter memory leak?
> > > > > >
> > > > > > Yes, that's the trimBuffer version I was thinking about, only
> > > this guy
> > > > > > created a reset(Reader, int) and does both ops (resetting +
> trim)
> > > in
> > > > > > one
> > > > > > method call. More convenient. Can you please open an issue to
> > > track
> > > > > > that?
> > > > > > People will have a chance to comment on whether we (Lucene)
> > > should
> > > > > > handle
> > > > > > that, or it should be a JFlex fix. Based on the number of
> replies
> > > this
> > > > > > guy
> > > > > > received (0 !), I doubt JFlex would consider it a problem.
> But we
> > > can
> > > > > > do
> > > > > > some small service to our users base by protecting against
> such
> > > > > > problems.
> > > > > >
> > > > > > And while you're opening the issue, if you want to take a
> stab at
> > > > > > fixing it
> > > > > > and post a patch, it'd be great :).
> > > > > >
> > > > > > Shai
> > > > > >
> > > > > > On Thu, Apr 8, 2010 at 12:51 PM, Ruben Laguna
> > > > > > <ru...@gmail.com>wrote:
> > > > > >
> > > > > > > I was investigating this a little further and in the JFlex
> > > mailing
> > > > > > list I
> > > > > > > found [1]
> > > > > > >
> > > > > > > I don't know much about flex / JFlex but it seems that this
> guy
> > > > > > resets the
> > > > > > > zzBuffer to 16384 or less when setting the input for the
> lexer
> > > > > > >
> > > > > > >
> > > > > > > Quoted from shef <sh...@ya...>
> > > > > > >
> > > > > > >
> > > > > > > I set
> > > > > > >
> > > > > > > %buffer 0
> > > > > > >
> > > > > > > in the options section, and then added this method to the
> > > lexer:
> > > > > > >
> > > > > > > /**
> > > > > > > * Set the input for the lexer. The size parameter
> really
> > > speeds
> > > > > > things
> > > > > > > up,
> > > > > > > * because by default, the lexer allocates an internal
> > > buffer of
> > > > > > 16k.
> > > > > > > For
> > > > > > > * most strings, this is unnecessarily large. If the
> size
> > > param is
> > > > > > > 0 or greater
> > > > > > > * than 16k, then the buffer is set to 16k. If the size
> > > param is
> > > > > > > smaller, then
> > > > > > > * the buf will be set to the exact size.
> > > > > > > * @param r the reader that provides the data
> > > > > > > * @param the size of the data in the reader.
> > > > > > > */
> > > > > > > public void reset(Reader r, int size) {
> > > > > > > if (size == 0 || size > 16384)
> > > > > > > size = 16384;
> > > > > > > zzBuffer = new char[size];
> > > > > > > yyreset(r);
> > > > > > > }
> > > > > > >
> > > > > > >
> > > > > > > So maybe there is a way to trim the zzBuffer this way (?).
> > > > > > >
> > > > > > > BTW, I will try to find out which is the "big token" in my
> > > dataset
> > > > > > this
> > > > > > > afternoon. Thanks for the help.
> > > > > > >
> > > > > > > I actually workaround this memory problem for the time
> being by
> > > > > > wrapping
> > > > > > > the
> > > > > > > IndexWriter in a class that periodically closes the
> IndexWriter
> > > and
> > > > > > creates
> > > > > > > a new one, allowing the old to be GCed, but I would be
> really
> > > good if
> > > > > > > either
> > > > > > > JFlex or Lucene can take care of this zzBuffer going
> berserk.
> > > > > > >
> > > > > > >
> > > > > > > Again thanks for the quick response. /Rubén
> > > > > > >
> > > > > > >
> > > > > > > [1]
> > > > > > >
> > > > > > >
> > > > > >
> > > >
> > >
> https://sourceforge.net/mailarchive/message.php?msg_id=444070.38422.qm@
> > > > > > web38901.mail.mud.yahoo.com
> > > > > > >
> > > > > > > On Thu, Apr 8, 2010 at 11:32 AM, Shai Erera
> <se...@gmail.com>
> > > > wrote:
> > > > > > >
> > > > > > > > If we could change the Flex file so that yyreset(Reader)
> > > would
> > > > > > check the
> > > > > > > > size of zzBuffer, we could trim it when it gets too big.
> But
> > > I
> > > > > > don't
> > > > > > > think
> > > > > > > > we have such control when writing the flex syntax ...
> yyreset
> > > is
> > > > > > > generated
> > > > > > > > by JFlex and that's the only place I can think of to trim
> the
> > > > > > buffer down
> > > > > > > > when it exceeds a predefined threshold ....
> > > > > > > >
> > > > > > > > Maybe what we can do is create our own method which will
> be
> > > called
> > > > > > by
> > > > > > > > StandardTokenizer after yyreset is called, something like
> > > > > > > > trimBufferIfTooBig(int threshold) which will reallocate
> > > zzBuffer if
> > > > > > it
> > > > > > > > exceeded the threshold. We can decide on a reasonable 64K
> > > threshold
> > > > > > or
> > > > > > > > something, or simply always cut back to 16 KB. As far as
> I
> > > > > > understand,
> > > > > > > that
> > > > > > > > buffer should never grow that much. I.e. in zzRefill,
> which
> > > is the
> > > > > > only
> > > > > > > > place where the buffer gets resized, there is an attempt
> to
> > > first
> > > > > > move
> > > > > > > back
> > > > > > > > characters that were already consumed and only then
> allocate
> > > a
> > > > > > bigger
> > > > > > > > buffer. Which means only if there is a token whose size
> is
> > > larger
> > > > > > than
> > > > > > > 16KB
> > > > > > > > (!?), will this buffer get expanded.
> > > > > > > >
> > > > > > > > A trimBuffer method might not be that bad .. as a
> protective
> > > > > > measure.
> > > > > > > What
> > > > > > > > do you think? Of course, JFlex can fix it on their own
> ...
> > > but
> > > > > > until that
> > > > > > > > happens ...
> > > > > > > >
> > > > > > > > Shai
> > > > > > > >
> > > > > > > > On Thu, Apr 8, 2010 at 10:35 AM, Uwe Schindler
> > > <uw...@thetaphi.de>
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > > I would like to identify also the problematic
> document I
> > > have
> > > > > > 10000
> > > > > > > so,
> > > > > > > > > > what
> > > > > > > > > > would be the best way of identifying the one that it
> > > making
> > > > > > zzBuffer
> > > > > > > to
> > > > > > > > > > grow
> > > > > > > > > > without control?
> > > > > > > > >
> > > > > > > > > Dont index your documents, but instead pass them
> directly
> > > to the
> > > > > > > analyzer
> > > > > > > > > and consume the tokenstream manually. Then visit
> > > > > > > > TermAttribute.termLength()
> > > > > > > > > for each Token.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > -------------------------------------------------------
> ----
> > > ------
> > > > > > ----
> > > > > > > > > To unsubscribe, e-mail: java-user-
> > > unsubscribe@lucene.apache.org
> > > > > > > > > For additional commands, e-mail:
> > > > java-user-help@lucene.apache.org
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > /Rubén
> > > > > > >
> > > > >
> > > > >
> > > > > ---------------------------------------------------------------
> ----
> > > --
> > > > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > > > > For additional commands, e-mail: java-user-
> help@lucene.apache.org
> > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > /Rubén
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
>
>
> --
> /Rubén
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: IndexWriter memory leak?
Posted by Uwe Schindler <uw...@thetaphi.de>.
There is one possibility, that could be fixed:
As Tokenizers are reused, the analyzer holds a reference to the last used Reader. The easy fix would be to unset the Reader in Tokenizer.close(). If this is the case for you, that may be easy to do. So Tokenizer.close() looks like this:
/** By default, closes the input Reader. */
@Override
public void close() throws IOException {
input.close();
input = null; // <-- new!
}
Any comments from other committers?
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: Ruben Laguna [mailto:ruben.laguna@gmail.com]
> Sent: Thursday, April 08, 2010 2:50 PM
> To: java-user@lucene.apache.org
> Subject: Re: IndexWriter memory leak?
>
> I will double check in the afternoon the heapdump.hprof. But I think
> that
> *some* readers are indeed held by
> docWriter.threadStates[0].consumer.fieldHash[1].fields[xxxx],
> as shown in [1] (this heapdump contains only live objects). The
> heapdump
> was taken after IndexWriter.commit() /IndexWriter.optimize() and all
> the
> Documents were already indexed and GCed (I will double check).
>
> So that would mean that the Reader is retained in memory by the
> following
> chaing of references,
>
> DocumentsWriter -> DocumentsWriterThreadState ->
> DocFieldProcessorPerThread
> -> DocFieldProcessorPerField -> Fieldable -> Field (fieldsData)
>
> I'll double check with Eclipse MAT as I said that this chain is
> actually
> made of hard references only (no SoftReferences,WeakReferences, etc). I
> will
> also double check also that there is no "live" Document that is
> referencing
> the Reader via the Field.
>
>
> [1] http://img.skitch.com/20100407-b86irkp7e4uif2wq1dd4t899qb.jpg
>
> On Thu, Apr 8, 2010 at 2:16 PM, Uwe Schindler <uw...@thetaphi.de> wrote:
>
> > Readers are not held. If you indexed the document and gced the
> document
> > instance they readers are gone.
> >
> > -----
> > Uwe Schindler
> > H.-H.-Meier-Allee 63, D-28213 Bremen
> > http://www.thetaphi.de
> > eMail: uwe@thetaphi.de
> >
> >
> > > -----Original Message-----
> > > From: Ruben Laguna [mailto:ruben.laguna@gmail.com]
> > > Sent: Thursday, April 08, 2010 1:28 PM
> > > To: java-user@lucene.apache.org
> > > Subject: Re: IndexWriter memory leak?
> > >
> > > Now that the zzBuffer issue is solved...
> > >
> > > what about the references to the Readers held by docWriter. Tika´s
> > > ParsingReaders are quite heavyweight so retaining those in memory
> > > unnecesarily is also a "hidden" memory leak. Should I open a bug
> report
> > > on
> > > that one?
> > >
> > > /Rubén
> > >
> > > On Thu, Apr 8, 2010 at 12:11 PM, Shai Erera <se...@gmail.com>
> wrote:
> > >
> > > > Guess we were replying at the same time :).
> > > >
> > > > On Thu, Apr 8, 2010 at 1:04 PM, Uwe Schindler <uw...@thetaphi.de>
> > > wrote:
> > > >
> > > > > I already answered, that I will take care of this!
> > > > >
> > > > > Uwe
> > > > >
> > > > > -----
> > > > > Uwe Schindler
> > > > > H.-H.-Meier-Allee 63, D-28213 Bremen
> > > > > http://www.thetaphi.de
> > > > > eMail: uwe@thetaphi.de
> > > > >
> > > > >
> > > > > > -----Original Message-----
> > > > > > From: Shai Erera [mailto:serera@gmail.com]
> > > > > > Sent: Thursday, April 08, 2010 12:00 PM
> > > > > > To: java-user@lucene.apache.org
> > > > > > Subject: Re: IndexWriter memory leak?
> > > > > >
> > > > > > Yes, that's the trimBuffer version I was thinking about, only
> > > this guy
> > > > > > created a reset(Reader, int) and does both ops (resetting +
> trim)
> > > in
> > > > > > one
> > > > > > method call. More convenient. Can you please open an issue to
> > > track
> > > > > > that?
> > > > > > People will have a chance to comment on whether we (Lucene)
> > > should
> > > > > > handle
> > > > > > that, or it should be a JFlex fix. Based on the number of
> replies
> > > this
> > > > > > guy
> > > > > > received (0 !), I doubt JFlex would consider it a problem.
> But we
> > > can
> > > > > > do
> > > > > > some small service to our users base by protecting against
> such
> > > > > > problems.
> > > > > >
> > > > > > And while you're opening the issue, if you want to take a
> stab at
> > > > > > fixing it
> > > > > > and post a patch, it'd be great :).
> > > > > >
> > > > > > Shai
> > > > > >
> > > > > > On Thu, Apr 8, 2010 at 12:51 PM, Ruben Laguna
> > > > > > <ru...@gmail.com>wrote:
> > > > > >
> > > > > > > I was investigating this a little further and in the JFlex
> > > mailing
> > > > > > list I
> > > > > > > found [1]
> > > > > > >
> > > > > > > I don't know much about flex / JFlex but it seems that this
> guy
> > > > > > resets the
> > > > > > > zzBuffer to 16384 or less when setting the input for the
> lexer
> > > > > > >
> > > > > > >
> > > > > > > Quoted from shef <sh...@ya...>
> > > > > > >
> > > > > > >
> > > > > > > I set
> > > > > > >
> > > > > > > %buffer 0
> > > > > > >
> > > > > > > in the options section, and then added this method to the
> > > lexer:
> > > > > > >
> > > > > > > /**
> > > > > > > * Set the input for the lexer. The size parameter
> really
> > > speeds
> > > > > > things
> > > > > > > up,
> > > > > > > * because by default, the lexer allocates an internal
> > > buffer of
> > > > > > 16k.
> > > > > > > For
> > > > > > > * most strings, this is unnecessarily large. If the
> size
> > > param is
> > > > > > > 0 or greater
> > > > > > > * than 16k, then the buffer is set to 16k. If the size
> > > param is
> > > > > > > smaller, then
> > > > > > > * the buf will be set to the exact size.
> > > > > > > * @param r the reader that provides the data
> > > > > > > * @param the size of the data in the reader.
> > > > > > > */
> > > > > > > public void reset(Reader r, int size) {
> > > > > > > if (size == 0 || size > 16384)
> > > > > > > size = 16384;
> > > > > > > zzBuffer = new char[size];
> > > > > > > yyreset(r);
> > > > > > > }
> > > > > > >
> > > > > > >
> > > > > > > So maybe there is a way to trim the zzBuffer this way (?).
> > > > > > >
> > > > > > > BTW, I will try to find out which is the "big token" in my
> > > dataset
> > > > > > this
> > > > > > > afternoon. Thanks for the help.
> > > > > > >
> > > > > > > I actually workaround this memory problem for the time
> being by
> > > > > > wrapping
> > > > > > > the
> > > > > > > IndexWriter in a class that periodically closes the
> IndexWriter
> > > and
> > > > > > creates
> > > > > > > a new one, allowing the old to be GCed, but I would be
> really
> > > good if
> > > > > > > either
> > > > > > > JFlex or Lucene can take care of this zzBuffer going
> berserk.
> > > > > > >
> > > > > > >
> > > > > > > Again thanks for the quick response. /Rubén
> > > > > > >
> > > > > > >
> > > > > > > [1]
> > > > > > >
> > > > > > >
> > > > > >
> > > >
> > >
> https://sourceforge.net/mailarchive/message.php?msg_id=444070.38422.qm@
> > > > > > web38901.mail.mud.yahoo.com
> > > > > > >
> > > > > > > On Thu, Apr 8, 2010 at 11:32 AM, Shai Erera
> <se...@gmail.com>
> > > > wrote:
> > > > > > >
> > > > > > > > If we could change the Flex file so that yyreset(Reader)
> > > would
> > > > > > check the
> > > > > > > > size of zzBuffer, we could trim it when it gets too big.
> But
> > > I
> > > > > > don't
> > > > > > > think
> > > > > > > > we have such control when writing the flex syntax ...
> yyreset
> > > is
> > > > > > > generated
> > > > > > > > by JFlex and that's the only place I can think of to trim
> the
> > > > > > buffer down
> > > > > > > > when it exceeds a predefined threshold ....
> > > > > > > >
> > > > > > > > Maybe what we can do is create our own method which will
> be
> > > called
> > > > > > by
> > > > > > > > StandardTokenizer after yyreset is called, something like
> > > > > > > > trimBufferIfTooBig(int threshold) which will reallocate
> > > zzBuffer if
> > > > > > it
> > > > > > > > exceeded the threshold. We can decide on a reasonable 64K
> > > threshold
> > > > > > or
> > > > > > > > something, or simply always cut back to 16 KB. As far as
> I
> > > > > > understand,
> > > > > > > that
> > > > > > > > buffer should never grow that much. I.e. in zzRefill,
> which
> > > is the
> > > > > > only
> > > > > > > > place where the buffer gets resized, there is an attempt
> to
> > > first
> > > > > > move
> > > > > > > back
> > > > > > > > characters that were already consumed and only then
> allocate
> > > a
> > > > > > bigger
> > > > > > > > buffer. Which means only if there is a token whose size
> is
> > > larger
> > > > > > than
> > > > > > > 16KB
> > > > > > > > (!?), will this buffer get expanded.
> > > > > > > >
> > > > > > > > A trimBuffer method might not be that bad .. as a
> protective
> > > > > > measure.
> > > > > > > What
> > > > > > > > do you think? Of course, JFlex can fix it on their own
> ...
> > > but
> > > > > > until that
> > > > > > > > happens ...
> > > > > > > >
> > > > > > > > Shai
> > > > > > > >
> > > > > > > > On Thu, Apr 8, 2010 at 10:35 AM, Uwe Schindler
> > > <uw...@thetaphi.de>
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > > I would like to identify also the problematic
> document I
> > > have
> > > > > > 10000
> > > > > > > so,
> > > > > > > > > > what
> > > > > > > > > > would be the best way of identifying the one that it
> > > making
> > > > > > zzBuffer
> > > > > > > to
> > > > > > > > > > grow
> > > > > > > > > > without control?
> > > > > > > > >
> > > > > > > > > Dont index your documents, but instead pass them
> directly
> > > to the
> > > > > > > analyzer
> > > > > > > > > and consume the tokenstream manually. Then visit
> > > > > > > > TermAttribute.termLength()
> > > > > > > > > for each Token.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > -------------------------------------------------------
> ----
> > > ------
> > > > > > ----
> > > > > > > > > To unsubscribe, e-mail: java-user-
> > > unsubscribe@lucene.apache.org
> > > > > > > > > For additional commands, e-mail:
> > > > java-user-help@lucene.apache.org
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > /Rubén
> > > > > > >
> > > > >
> > > > >
> > > > > ---------------------------------------------------------------
> ----
> > > --
> > > > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > > > > For additional commands, e-mail: java-user-
> help@lucene.apache.org
> > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > /Rubén
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
>
>
> --
> /Rubén
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: IndexWriter memory leak?
Posted by Ruben Laguna <ru...@gmail.com>.
I will double check in the afternoon the heapdump.hprof. But I think that
*some* readers are indeed held by
docWriter.threadStates[0].consumer.fieldHash[1].fields[xxxx],
as shown in [1] (this heapdump contains only live objects). The heapdump
was taken after IndexWriter.commit() /IndexWriter.optimize() and all the
Documents were already indexed and GCed (I will double check).
So that would mean that the Reader is retained in memory by the following
chaing of references,
DocumentsWriter -> DocumentsWriterThreadState -> DocFieldProcessorPerThread
-> DocFieldProcessorPerField -> Fieldable -> Field (fieldsData)
I'll double check with Eclipse MAT as I said that this chain is actually
made of hard references only (no SoftReferences,WeakReferences, etc). I will
also double check also that there is no "live" Document that is referencing
the Reader via the Field.
[1] http://img.skitch.com/20100407-b86irkp7e4uif2wq1dd4t899qb.jpg
On Thu, Apr 8, 2010 at 2:16 PM, Uwe Schindler <uw...@thetaphi.de> wrote:
> Readers are not held. If you indexed the document and gced the document
> instance they readers are gone.
>
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: uwe@thetaphi.de
>
>
> > -----Original Message-----
> > From: Ruben Laguna [mailto:ruben.laguna@gmail.com]
> > Sent: Thursday, April 08, 2010 1:28 PM
> > To: java-user@lucene.apache.org
> > Subject: Re: IndexWriter memory leak?
> >
> > Now that the zzBuffer issue is solved...
> >
> > what about the references to the Readers held by docWriter. Tika´s
> > ParsingReaders are quite heavyweight so retaining those in memory
> > unnecesarily is also a "hidden" memory leak. Should I open a bug report
> > on
> > that one?
> >
> > /Rubén
> >
> > On Thu, Apr 8, 2010 at 12:11 PM, Shai Erera <se...@gmail.com> wrote:
> >
> > > Guess we were replying at the same time :).
> > >
> > > On Thu, Apr 8, 2010 at 1:04 PM, Uwe Schindler <uw...@thetaphi.de>
> > wrote:
> > >
> > > > I already answered, that I will take care of this!
> > > >
> > > > Uwe
> > > >
> > > > -----
> > > > Uwe Schindler
> > > > H.-H.-Meier-Allee 63, D-28213 Bremen
> > > > http://www.thetaphi.de
> > > > eMail: uwe@thetaphi.de
> > > >
> > > >
> > > > > -----Original Message-----
> > > > > From: Shai Erera [mailto:serera@gmail.com]
> > > > > Sent: Thursday, April 08, 2010 12:00 PM
> > > > > To: java-user@lucene.apache.org
> > > > > Subject: Re: IndexWriter memory leak?
> > > > >
> > > > > Yes, that's the trimBuffer version I was thinking about, only
> > this guy
> > > > > created a reset(Reader, int) and does both ops (resetting + trim)
> > in
> > > > > one
> > > > > method call. More convenient. Can you please open an issue to
> > track
> > > > > that?
> > > > > People will have a chance to comment on whether we (Lucene)
> > should
> > > > > handle
> > > > > that, or it should be a JFlex fix. Based on the number of replies
> > this
> > > > > guy
> > > > > received (0 !), I doubt JFlex would consider it a problem. But we
> > can
> > > > > do
> > > > > some small service to our users base by protecting against such
> > > > > problems.
> > > > >
> > > > > And while you're opening the issue, if you want to take a stab at
> > > > > fixing it
> > > > > and post a patch, it'd be great :).
> > > > >
> > > > > Shai
> > > > >
> > > > > On Thu, Apr 8, 2010 at 12:51 PM, Ruben Laguna
> > > > > <ru...@gmail.com>wrote:
> > > > >
> > > > > > I was investigating this a little further and in the JFlex
> > mailing
> > > > > list I
> > > > > > found [1]
> > > > > >
> > > > > > I don't know much about flex / JFlex but it seems that this guy
> > > > > resets the
> > > > > > zzBuffer to 16384 or less when setting the input for the lexer
> > > > > >
> > > > > >
> > > > > > Quoted from shef <sh...@ya...>
> > > > > >
> > > > > >
> > > > > > I set
> > > > > >
> > > > > > %buffer 0
> > > > > >
> > > > > > in the options section, and then added this method to the
> > lexer:
> > > > > >
> > > > > > /**
> > > > > > * Set the input for the lexer. The size parameter really
> > speeds
> > > > > things
> > > > > > up,
> > > > > > * because by default, the lexer allocates an internal
> > buffer of
> > > > > 16k.
> > > > > > For
> > > > > > * most strings, this is unnecessarily large. If the size
> > param is
> > > > > > 0 or greater
> > > > > > * than 16k, then the buffer is set to 16k. If the size
> > param is
> > > > > > smaller, then
> > > > > > * the buf will be set to the exact size.
> > > > > > * @param r the reader that provides the data
> > > > > > * @param the size of the data in the reader.
> > > > > > */
> > > > > > public void reset(Reader r, int size) {
> > > > > > if (size == 0 || size > 16384)
> > > > > > size = 16384;
> > > > > > zzBuffer = new char[size];
> > > > > > yyreset(r);
> > > > > > }
> > > > > >
> > > > > >
> > > > > > So maybe there is a way to trim the zzBuffer this way (?).
> > > > > >
> > > > > > BTW, I will try to find out which is the "big token" in my
> > dataset
> > > > > this
> > > > > > afternoon. Thanks for the help.
> > > > > >
> > > > > > I actually workaround this memory problem for the time being by
> > > > > wrapping
> > > > > > the
> > > > > > IndexWriter in a class that periodically closes the IndexWriter
> > and
> > > > > creates
> > > > > > a new one, allowing the old to be GCed, but I would be really
> > good if
> > > > > > either
> > > > > > JFlex or Lucene can take care of this zzBuffer going berserk.
> > > > > >
> > > > > >
> > > > > > Again thanks for the quick response. /Rubén
> > > > > >
> > > > > >
> > > > > > [1]
> > > > > >
> > > > > >
> > > > >
> > >
> > https://sourceforge.net/mailarchive/message.php?msg_id=444070.38422.qm@
> > > > > web38901.mail.mud.yahoo.com
> > > > > >
> > > > > > On Thu, Apr 8, 2010 at 11:32 AM, Shai Erera <se...@gmail.com>
> > > wrote:
> > > > > >
> > > > > > > If we could change the Flex file so that yyreset(Reader)
> > would
> > > > > check the
> > > > > > > size of zzBuffer, we could trim it when it gets too big. But
> > I
> > > > > don't
> > > > > > think
> > > > > > > we have such control when writing the flex syntax ... yyreset
> > is
> > > > > > generated
> > > > > > > by JFlex and that's the only place I can think of to trim the
> > > > > buffer down
> > > > > > > when it exceeds a predefined threshold ....
> > > > > > >
> > > > > > > Maybe what we can do is create our own method which will be
> > called
> > > > > by
> > > > > > > StandardTokenizer after yyreset is called, something like
> > > > > > > trimBufferIfTooBig(int threshold) which will reallocate
> > zzBuffer if
> > > > > it
> > > > > > > exceeded the threshold. We can decide on a reasonable 64K
> > threshold
> > > > > or
> > > > > > > something, or simply always cut back to 16 KB. As far as I
> > > > > understand,
> > > > > > that
> > > > > > > buffer should never grow that much. I.e. in zzRefill, which
> > is the
> > > > > only
> > > > > > > place where the buffer gets resized, there is an attempt to
> > first
> > > > > move
> > > > > > back
> > > > > > > characters that were already consumed and only then allocate
> > a
> > > > > bigger
> > > > > > > buffer. Which means only if there is a token whose size is
> > larger
> > > > > than
> > > > > > 16KB
> > > > > > > (!?), will this buffer get expanded.
> > > > > > >
> > > > > > > A trimBuffer method might not be that bad .. as a protective
> > > > > measure.
> > > > > > What
> > > > > > > do you think? Of course, JFlex can fix it on their own ...
> > but
> > > > > until that
> > > > > > > happens ...
> > > > > > >
> > > > > > > Shai
> > > > > > >
> > > > > > > On Thu, Apr 8, 2010 at 10:35 AM, Uwe Schindler
> > <uw...@thetaphi.de>
> > > > > wrote:
> > > > > > >
> > > > > > > > > I would like to identify also the problematic document I
> > have
> > > > > 10000
> > > > > > so,
> > > > > > > > > what
> > > > > > > > > would be the best way of identifying the one that it
> > making
> > > > > zzBuffer
> > > > > > to
> > > > > > > > > grow
> > > > > > > > > without control?
> > > > > > > >
> > > > > > > > Dont index your documents, but instead pass them directly
> > to the
> > > > > > analyzer
> > > > > > > > and consume the tokenstream manually. Then visit
> > > > > > > TermAttribute.termLength()
> > > > > > > > for each Token.
> > > > > > > >
> > > > > > > >
> > > > > > > > -----------------------------------------------------------
> > ------
> > > > > ----
> > > > > > > > To unsubscribe, e-mail: java-user-
> > unsubscribe@lucene.apache.org
> > > > > > > > For additional commands, e-mail:
> > > java-user-help@lucene.apache.org
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > /Rubén
> > > > > >
> > > >
> > > >
> > > > -------------------------------------------------------------------
> > --
> > > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > > > For additional commands, e-mail: java-user-help@lucene.apache.org
> > > >
> > > >
> > >
> >
> >
> >
> > --
> > /Rubén
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
--
/Rubén
RE: IndexWriter memory leak?
Posted by Uwe Schindler <uw...@thetaphi.de>.
Readers are not held. If you indexed the document and gced the document instance they readers are gone.
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: Ruben Laguna [mailto:ruben.laguna@gmail.com]
> Sent: Thursday, April 08, 2010 1:28 PM
> To: java-user@lucene.apache.org
> Subject: Re: IndexWriter memory leak?
>
> Now that the zzBuffer issue is solved...
>
> what about the references to the Readers held by docWriter. Tika´s
> ParsingReaders are quite heavyweight so retaining those in memory
> unnecesarily is also a "hidden" memory leak. Should I open a bug report
> on
> that one?
>
> /Rubén
>
> On Thu, Apr 8, 2010 at 12:11 PM, Shai Erera <se...@gmail.com> wrote:
>
> > Guess we were replying at the same time :).
> >
> > On Thu, Apr 8, 2010 at 1:04 PM, Uwe Schindler <uw...@thetaphi.de>
> wrote:
> >
> > > I already answered, that I will take care of this!
> > >
> > > Uwe
> > >
> > > -----
> > > Uwe Schindler
> > > H.-H.-Meier-Allee 63, D-28213 Bremen
> > > http://www.thetaphi.de
> > > eMail: uwe@thetaphi.de
> > >
> > >
> > > > -----Original Message-----
> > > > From: Shai Erera [mailto:serera@gmail.com]
> > > > Sent: Thursday, April 08, 2010 12:00 PM
> > > > To: java-user@lucene.apache.org
> > > > Subject: Re: IndexWriter memory leak?
> > > >
> > > > Yes, that's the trimBuffer version I was thinking about, only
> this guy
> > > > created a reset(Reader, int) and does both ops (resetting + trim)
> in
> > > > one
> > > > method call. More convenient. Can you please open an issue to
> track
> > > > that?
> > > > People will have a chance to comment on whether we (Lucene)
> should
> > > > handle
> > > > that, or it should be a JFlex fix. Based on the number of replies
> this
> > > > guy
> > > > received (0 !), I doubt JFlex would consider it a problem. But we
> can
> > > > do
> > > > some small service to our users base by protecting against such
> > > > problems.
> > > >
> > > > And while you're opening the issue, if you want to take a stab at
> > > > fixing it
> > > > and post a patch, it'd be great :).
> > > >
> > > > Shai
> > > >
> > > > On Thu, Apr 8, 2010 at 12:51 PM, Ruben Laguna
> > > > <ru...@gmail.com>wrote:
> > > >
> > > > > I was investigating this a little further and in the JFlex
> mailing
> > > > list I
> > > > > found [1]
> > > > >
> > > > > I don't know much about flex / JFlex but it seems that this guy
> > > > resets the
> > > > > zzBuffer to 16384 or less when setting the input for the lexer
> > > > >
> > > > >
> > > > > Quoted from shef <sh...@ya...>
> > > > >
> > > > >
> > > > > I set
> > > > >
> > > > > %buffer 0
> > > > >
> > > > > in the options section, and then added this method to the
> lexer:
> > > > >
> > > > > /**
> > > > > * Set the input for the lexer. The size parameter really
> speeds
> > > > things
> > > > > up,
> > > > > * because by default, the lexer allocates an internal
> buffer of
> > > > 16k.
> > > > > For
> > > > > * most strings, this is unnecessarily large. If the size
> param is
> > > > > 0 or greater
> > > > > * than 16k, then the buffer is set to 16k. If the size
> param is
> > > > > smaller, then
> > > > > * the buf will be set to the exact size.
> > > > > * @param r the reader that provides the data
> > > > > * @param the size of the data in the reader.
> > > > > */
> > > > > public void reset(Reader r, int size) {
> > > > > if (size == 0 || size > 16384)
> > > > > size = 16384;
> > > > > zzBuffer = new char[size];
> > > > > yyreset(r);
> > > > > }
> > > > >
> > > > >
> > > > > So maybe there is a way to trim the zzBuffer this way (?).
> > > > >
> > > > > BTW, I will try to find out which is the "big token" in my
> dataset
> > > > this
> > > > > afternoon. Thanks for the help.
> > > > >
> > > > > I actually workaround this memory problem for the time being by
> > > > wrapping
> > > > > the
> > > > > IndexWriter in a class that periodically closes the IndexWriter
> and
> > > > creates
> > > > > a new one, allowing the old to be GCed, but I would be really
> good if
> > > > > either
> > > > > JFlex or Lucene can take care of this zzBuffer going berserk.
> > > > >
> > > > >
> > > > > Again thanks for the quick response. /Rubén
> > > > >
> > > > >
> > > > > [1]
> > > > >
> > > > >
> > > >
> >
> https://sourceforge.net/mailarchive/message.php?msg_id=444070.38422.qm@
> > > > web38901.mail.mud.yahoo.com
> > > > >
> > > > > On Thu, Apr 8, 2010 at 11:32 AM, Shai Erera <se...@gmail.com>
> > wrote:
> > > > >
> > > > > > If we could change the Flex file so that yyreset(Reader)
> would
> > > > check the
> > > > > > size of zzBuffer, we could trim it when it gets too big. But
> I
> > > > don't
> > > > > think
> > > > > > we have such control when writing the flex syntax ... yyreset
> is
> > > > > generated
> > > > > > by JFlex and that's the only place I can think of to trim the
> > > > buffer down
> > > > > > when it exceeds a predefined threshold ....
> > > > > >
> > > > > > Maybe what we can do is create our own method which will be
> called
> > > > by
> > > > > > StandardTokenizer after yyreset is called, something like
> > > > > > trimBufferIfTooBig(int threshold) which will reallocate
> zzBuffer if
> > > > it
> > > > > > exceeded the threshold. We can decide on a reasonable 64K
> threshold
> > > > or
> > > > > > something, or simply always cut back to 16 KB. As far as I
> > > > understand,
> > > > > that
> > > > > > buffer should never grow that much. I.e. in zzRefill, which
> is the
> > > > only
> > > > > > place where the buffer gets resized, there is an attempt to
> first
> > > > move
> > > > > back
> > > > > > characters that were already consumed and only then allocate
> a
> > > > bigger
> > > > > > buffer. Which means only if there is a token whose size is
> larger
> > > > than
> > > > > 16KB
> > > > > > (!?), will this buffer get expanded.
> > > > > >
> > > > > > A trimBuffer method might not be that bad .. as a protective
> > > > measure.
> > > > > What
> > > > > > do you think? Of course, JFlex can fix it on their own ...
> but
> > > > until that
> > > > > > happens ...
> > > > > >
> > > > > > Shai
> > > > > >
> > > > > > On Thu, Apr 8, 2010 at 10:35 AM, Uwe Schindler
> <uw...@thetaphi.de>
> > > > wrote:
> > > > > >
> > > > > > > > I would like to identify also the problematic document I
> have
> > > > 10000
> > > > > so,
> > > > > > > > what
> > > > > > > > would be the best way of identifying the one that it
> making
> > > > zzBuffer
> > > > > to
> > > > > > > > grow
> > > > > > > > without control?
> > > > > > >
> > > > > > > Dont index your documents, but instead pass them directly
> to the
> > > > > analyzer
> > > > > > > and consume the tokenstream manually. Then visit
> > > > > > TermAttribute.termLength()
> > > > > > > for each Token.
> > > > > > >
> > > > > > >
> > > > > > > -----------------------------------------------------------
> ------
> > > > ----
> > > > > > > To unsubscribe, e-mail: java-user-
> unsubscribe@lucene.apache.org
> > > > > > > For additional commands, e-mail:
> > java-user-help@lucene.apache.org
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > /Rubén
> > > > >
> > >
> > >
> > > -------------------------------------------------------------------
> --
> > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > > For additional commands, e-mail: java-user-help@lucene.apache.org
> > >
> > >
> >
>
>
>
> --
> /Rubén
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: IndexWriter memory leak?
Posted by Ruben Laguna <ru...@gmail.com>.
And by the way, when is Lucene 3.1 coming?
On Thu, Apr 8, 2010 at 1:27 PM, Ruben Laguna <ru...@gmail.com> wrote:
> Now that the zzBuffer issue is solved...
>
> what about the references to the Readers held by docWriter. Tika´s
> ParsingReaders are quite heavyweight so retaining those in memory
> unnecesarily is also a "hidden" memory leak. Should I open a bug report on
> that one?
>
> /Rubén
>
> On Thu, Apr 8, 2010 at 12:11 PM, Shai Erera <se...@gmail.com> wrote:
>
>> Guess we were replying at the same time :).
>>
>> On Thu, Apr 8, 2010 at 1:04 PM, Uwe Schindler <uw...@thetaphi.de> wrote:
>>
>> > I already answered, that I will take care of this!
>> >
>> > Uwe
>> >
>> > -----
>> > Uwe Schindler
>> > H.-H.-Meier-Allee 63, D-28213 Bremen
>> > http://www.thetaphi.de
>> > eMail: uwe@thetaphi.de
>> >
>> >
>> > > -----Original Message-----
>> > > From: Shai Erera [mailto:serera@gmail.com]
>> > > Sent: Thursday, April 08, 2010 12:00 PM
>> > > To: java-user@lucene.apache.org
>> > > Subject: Re: IndexWriter memory leak?
>> > >
>> > > Yes, that's the trimBuffer version I was thinking about, only this guy
>> > > created a reset(Reader, int) and does both ops (resetting + trim) in
>> > > one
>> > > method call. More convenient. Can you please open an issue to track
>> > > that?
>> > > People will have a chance to comment on whether we (Lucene) should
>> > > handle
>> > > that, or it should be a JFlex fix. Based on the number of replies this
>> > > guy
>> > > received (0 !), I doubt JFlex would consider it a problem. But we can
>> > > do
>> > > some small service to our users base by protecting against such
>> > > problems.
>> > >
>> > > And while you're opening the issue, if you want to take a stab at
>> > > fixing it
>> > > and post a patch, it'd be great :).
>> > >
>> > > Shai
>> > >
>> > > On Thu, Apr 8, 2010 at 12:51 PM, Ruben Laguna
>> > > <ru...@gmail.com>wrote:
>> > >
>> > > > I was investigating this a little further and in the JFlex mailing
>> > > list I
>> > > > found [1]
>> > > >
>> > > > I don't know much about flex / JFlex but it seems that this guy
>> > > resets the
>> > > > zzBuffer to 16384 or less when setting the input for the lexer
>> > > >
>> > > >
>> > > > Quoted from shef <sh...@ya...>
>> > > >
>> > > >
>> > > > I set
>> > > >
>> > > > %buffer 0
>> > > >
>> > > > in the options section, and then added this method to the lexer:
>> > > >
>> > > > /**
>> > > > * Set the input for the lexer. The size parameter really speeds
>> > > things
>> > > > up,
>> > > > * because by default, the lexer allocates an internal buffer of
>> > > 16k.
>> > > > For
>> > > > * most strings, this is unnecessarily large. If the size param
>> is
>> > > > 0 or greater
>> > > > * than 16k, then the buffer is set to 16k. If the size param is
>> > > > smaller, then
>> > > > * the buf will be set to the exact size.
>> > > > * @param r the reader that provides the data
>> > > > * @param the size of the data in the reader.
>> > > > */
>> > > > public void reset(Reader r, int size) {
>> > > > if (size == 0 || size > 16384)
>> > > > size = 16384;
>> > > > zzBuffer = new char[size];
>> > > > yyreset(r);
>> > > > }
>> > > >
>> > > >
>> > > > So maybe there is a way to trim the zzBuffer this way (?).
>> > > >
>> > > > BTW, I will try to find out which is the "big token" in my dataset
>> > > this
>> > > > afternoon. Thanks for the help.
>> > > >
>> > > > I actually workaround this memory problem for the time being by
>> > > wrapping
>> > > > the
>> > > > IndexWriter in a class that periodically closes the IndexWriter and
>> > > creates
>> > > > a new one, allowing the old to be GCed, but I would be really good
>> if
>> > > > either
>> > > > JFlex or Lucene can take care of this zzBuffer going berserk.
>> > > >
>> > > >
>> > > > Again thanks for the quick response. /Rubén
>> > > >
>> > > >
>> > > > [1]
>> > > >
>> > > >
>> > >
>> https://sourceforge.net/mailarchive/message.php?msg_id=444070.38422.qm@
>> > > web38901.mail.mud.yahoo.com
>> > > >
>> > > > On Thu, Apr 8, 2010 at 11:32 AM, Shai Erera <se...@gmail.com>
>> wrote:
>> > > >
>> > > > > If we could change the Flex file so that yyreset(Reader) would
>> > > check the
>> > > > > size of zzBuffer, we could trim it when it gets too big. But I
>> > > don't
>> > > > think
>> > > > > we have such control when writing the flex syntax ... yyreset is
>> > > > generated
>> > > > > by JFlex and that's the only place I can think of to trim the
>> > > buffer down
>> > > > > when it exceeds a predefined threshold ....
>> > > > >
>> > > > > Maybe what we can do is create our own method which will be called
>> > > by
>> > > > > StandardTokenizer after yyreset is called, something like
>> > > > > trimBufferIfTooBig(int threshold) which will reallocate zzBuffer
>> if
>> > > it
>> > > > > exceeded the threshold. We can decide on a reasonable 64K
>> threshold
>> > > or
>> > > > > something, or simply always cut back to 16 KB. As far as I
>> > > understand,
>> > > > that
>> > > > > buffer should never grow that much. I.e. in zzRefill, which is the
>> > > only
>> > > > > place where the buffer gets resized, there is an attempt to first
>> > > move
>> > > > back
>> > > > > characters that were already consumed and only then allocate a
>> > > bigger
>> > > > > buffer. Which means only if there is a token whose size is larger
>> > > than
>> > > > 16KB
>> > > > > (!?), will this buffer get expanded.
>> > > > >
>> > > > > A trimBuffer method might not be that bad .. as a protective
>> > > measure.
>> > > > What
>> > > > > do you think? Of course, JFlex can fix it on their own ... but
>> > > until that
>> > > > > happens ...
>> > > > >
>> > > > > Shai
>> > > > >
>> > > > > On Thu, Apr 8, 2010 at 10:35 AM, Uwe Schindler <uw...@thetaphi.de>
>> > > wrote:
>> > > > >
>> > > > > > > I would like to identify also the problematic document I have
>> > > 10000
>> > > > so,
>> > > > > > > what
>> > > > > > > would be the best way of identifying the one that it making
>> > > zzBuffer
>> > > > to
>> > > > > > > grow
>> > > > > > > without control?
>> > > > > >
>> > > > > > Dont index your documents, but instead pass them directly to the
>> > > > analyzer
>> > > > > > and consume the tokenstream manually. Then visit
>> > > > > TermAttribute.termLength()
>> > > > > > for each Token.
>> > > > > >
>> > > > > >
>> > > > > >
>> -----------------------------------------------------------------
>> > > ----
>> > > > > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> > > > > > For additional commands, e-mail:
>> java-user-help@lucene.apache.org
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > > >
>> > > >
>> > > > --
>> > > > /Rubén
>> > > >
>> >
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> > For additional commands, e-mail: java-user-help@lucene.apache.org
>> >
>> >
>>
>
>
>
> --
> /Rubén
>
--
/Rubén
Re: IndexWriter memory leak?
Posted by Ruben Laguna <ru...@gmail.com>.
Now that the zzBuffer issue is solved...
what about the references to the Readers held by docWriter. Tika´s
ParsingReaders are quite heavyweight so retaining those in memory
unnecesarily is also a "hidden" memory leak. Should I open a bug report on
that one?
/Rubén
On Thu, Apr 8, 2010 at 12:11 PM, Shai Erera <se...@gmail.com> wrote:
> Guess we were replying at the same time :).
>
> On Thu, Apr 8, 2010 at 1:04 PM, Uwe Schindler <uw...@thetaphi.de> wrote:
>
> > I already answered, that I will take care of this!
> >
> > Uwe
> >
> > -----
> > Uwe Schindler
> > H.-H.-Meier-Allee 63, D-28213 Bremen
> > http://www.thetaphi.de
> > eMail: uwe@thetaphi.de
> >
> >
> > > -----Original Message-----
> > > From: Shai Erera [mailto:serera@gmail.com]
> > > Sent: Thursday, April 08, 2010 12:00 PM
> > > To: java-user@lucene.apache.org
> > > Subject: Re: IndexWriter memory leak?
> > >
> > > Yes, that's the trimBuffer version I was thinking about, only this guy
> > > created a reset(Reader, int) and does both ops (resetting + trim) in
> > > one
> > > method call. More convenient. Can you please open an issue to track
> > > that?
> > > People will have a chance to comment on whether we (Lucene) should
> > > handle
> > > that, or it should be a JFlex fix. Based on the number of replies this
> > > guy
> > > received (0 !), I doubt JFlex would consider it a problem. But we can
> > > do
> > > some small service to our users base by protecting against such
> > > problems.
> > >
> > > And while you're opening the issue, if you want to take a stab at
> > > fixing it
> > > and post a patch, it'd be great :).
> > >
> > > Shai
> > >
> > > On Thu, Apr 8, 2010 at 12:51 PM, Ruben Laguna
> > > <ru...@gmail.com>wrote:
> > >
> > > > I was investigating this a little further and in the JFlex mailing
> > > list I
> > > > found [1]
> > > >
> > > > I don't know much about flex / JFlex but it seems that this guy
> > > resets the
> > > > zzBuffer to 16384 or less when setting the input for the lexer
> > > >
> > > >
> > > > Quoted from shef <sh...@ya...>
> > > >
> > > >
> > > > I set
> > > >
> > > > %buffer 0
> > > >
> > > > in the options section, and then added this method to the lexer:
> > > >
> > > > /**
> > > > * Set the input for the lexer. The size parameter really speeds
> > > things
> > > > up,
> > > > * because by default, the lexer allocates an internal buffer of
> > > 16k.
> > > > For
> > > > * most strings, this is unnecessarily large. If the size param is
> > > > 0 or greater
> > > > * than 16k, then the buffer is set to 16k. If the size param is
> > > > smaller, then
> > > > * the buf will be set to the exact size.
> > > > * @param r the reader that provides the data
> > > > * @param the size of the data in the reader.
> > > > */
> > > > public void reset(Reader r, int size) {
> > > > if (size == 0 || size > 16384)
> > > > size = 16384;
> > > > zzBuffer = new char[size];
> > > > yyreset(r);
> > > > }
> > > >
> > > >
> > > > So maybe there is a way to trim the zzBuffer this way (?).
> > > >
> > > > BTW, I will try to find out which is the "big token" in my dataset
> > > this
> > > > afternoon. Thanks for the help.
> > > >
> > > > I actually workaround this memory problem for the time being by
> > > wrapping
> > > > the
> > > > IndexWriter in a class that periodically closes the IndexWriter and
> > > creates
> > > > a new one, allowing the old to be GCed, but I would be really good if
> > > > either
> > > > JFlex or Lucene can take care of this zzBuffer going berserk.
> > > >
> > > >
> > > > Again thanks for the quick response. /Rubén
> > > >
> > > >
> > > > [1]
> > > >
> > > >
> > >
> https://sourceforge.net/mailarchive/message.php?msg_id=444070.38422.qm@
> > > web38901.mail.mud.yahoo.com
> > > >
> > > > On Thu, Apr 8, 2010 at 11:32 AM, Shai Erera <se...@gmail.com>
> wrote:
> > > >
> > > > > If we could change the Flex file so that yyreset(Reader) would
> > > check the
> > > > > size of zzBuffer, we could trim it when it gets too big. But I
> > > don't
> > > > think
> > > > > we have such control when writing the flex syntax ... yyreset is
> > > > generated
> > > > > by JFlex and that's the only place I can think of to trim the
> > > buffer down
> > > > > when it exceeds a predefined threshold ....
> > > > >
> > > > > Maybe what we can do is create our own method which will be called
> > > by
> > > > > StandardTokenizer after yyreset is called, something like
> > > > > trimBufferIfTooBig(int threshold) which will reallocate zzBuffer if
> > > it
> > > > > exceeded the threshold. We can decide on a reasonable 64K threshold
> > > or
> > > > > something, or simply always cut back to 16 KB. As far as I
> > > understand,
> > > > that
> > > > > buffer should never grow that much. I.e. in zzRefill, which is the
> > > only
> > > > > place where the buffer gets resized, there is an attempt to first
> > > move
> > > > back
> > > > > characters that were already consumed and only then allocate a
> > > bigger
> > > > > buffer. Which means only if there is a token whose size is larger
> > > than
> > > > 16KB
> > > > > (!?), will this buffer get expanded.
> > > > >
> > > > > A trimBuffer method might not be that bad .. as a protective
> > > measure.
> > > > What
> > > > > do you think? Of course, JFlex can fix it on their own ... but
> > > until that
> > > > > happens ...
> > > > >
> > > > > Shai
> > > > >
> > > > > On Thu, Apr 8, 2010 at 10:35 AM, Uwe Schindler <uw...@thetaphi.de>
> > > wrote:
> > > > >
> > > > > > > I would like to identify also the problematic document I have
> > > 10000
> > > > so,
> > > > > > > what
> > > > > > > would be the best way of identifying the one that it making
> > > zzBuffer
> > > > to
> > > > > > > grow
> > > > > > > without control?
> > > > > >
> > > > > > Dont index your documents, but instead pass them directly to the
> > > > analyzer
> > > > > > and consume the tokenstream manually. Then visit
> > > > > TermAttribute.termLength()
> > > > > > for each Token.
> > > > > >
> > > > > >
> > > > > > -----------------------------------------------------------------
> > > ----
> > > > > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > > > > > For additional commands, e-mail:
> java-user-help@lucene.apache.org
> > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > /Rubén
> > > >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
>
--
/Rubén
Re: IndexWriter memory leak?
Posted by Shai Erera <se...@gmail.com>.
Guess we were replying at the same time :).
On Thu, Apr 8, 2010 at 1:04 PM, Uwe Schindler <uw...@thetaphi.de> wrote:
> I already answered, that I will take care of this!
>
> Uwe
>
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: uwe@thetaphi.de
>
>
> > -----Original Message-----
> > From: Shai Erera [mailto:serera@gmail.com]
> > Sent: Thursday, April 08, 2010 12:00 PM
> > To: java-user@lucene.apache.org
> > Subject: Re: IndexWriter memory leak?
> >
> > Yes, that's the trimBuffer version I was thinking about, only this guy
> > created a reset(Reader, int) and does both ops (resetting + trim) in
> > one
> > method call. More convenient. Can you please open an issue to track
> > that?
> > People will have a chance to comment on whether we (Lucene) should
> > handle
> > that, or it should be a JFlex fix. Based on the number of replies this
> > guy
> > received (0 !), I doubt JFlex would consider it a problem. But we can
> > do
> > some small service to our users base by protecting against such
> > problems.
> >
> > And while you're opening the issue, if you want to take a stab at
> > fixing it
> > and post a patch, it'd be great :).
> >
> > Shai
> >
> > On Thu, Apr 8, 2010 at 12:51 PM, Ruben Laguna
> > <ru...@gmail.com>wrote:
> >
> > > I was investigating this a little further and in the JFlex mailing
> > list I
> > > found [1]
> > >
> > > I don't know much about flex / JFlex but it seems that this guy
> > resets the
> > > zzBuffer to 16384 or less when setting the input for the lexer
> > >
> > >
> > > Quoted from shef <sh...@ya...>
> > >
> > >
> > > I set
> > >
> > > %buffer 0
> > >
> > > in the options section, and then added this method to the lexer:
> > >
> > > /**
> > > * Set the input for the lexer. The size parameter really speeds
> > things
> > > up,
> > > * because by default, the lexer allocates an internal buffer of
> > 16k.
> > > For
> > > * most strings, this is unnecessarily large. If the size param is
> > > 0 or greater
> > > * than 16k, then the buffer is set to 16k. If the size param is
> > > smaller, then
> > > * the buf will be set to the exact size.
> > > * @param r the reader that provides the data
> > > * @param the size of the data in the reader.
> > > */
> > > public void reset(Reader r, int size) {
> > > if (size == 0 || size > 16384)
> > > size = 16384;
> > > zzBuffer = new char[size];
> > > yyreset(r);
> > > }
> > >
> > >
> > > So maybe there is a way to trim the zzBuffer this way (?).
> > >
> > > BTW, I will try to find out which is the "big token" in my dataset
> > this
> > > afternoon. Thanks for the help.
> > >
> > > I actually workaround this memory problem for the time being by
> > wrapping
> > > the
> > > IndexWriter in a class that periodically closes the IndexWriter and
> > creates
> > > a new one, allowing the old to be GCed, but I would be really good if
> > > either
> > > JFlex or Lucene can take care of this zzBuffer going berserk.
> > >
> > >
> > > Again thanks for the quick response. /Rubén
> > >
> > >
> > > [1]
> > >
> > >
> > https://sourceforge.net/mailarchive/message.php?msg_id=444070.38422.qm@
> > web38901.mail.mud.yahoo.com
> > >
> > > On Thu, Apr 8, 2010 at 11:32 AM, Shai Erera <se...@gmail.com> wrote:
> > >
> > > > If we could change the Flex file so that yyreset(Reader) would
> > check the
> > > > size of zzBuffer, we could trim it when it gets too big. But I
> > don't
> > > think
> > > > we have such control when writing the flex syntax ... yyreset is
> > > generated
> > > > by JFlex and that's the only place I can think of to trim the
> > buffer down
> > > > when it exceeds a predefined threshold ....
> > > >
> > > > Maybe what we can do is create our own method which will be called
> > by
> > > > StandardTokenizer after yyreset is called, something like
> > > > trimBufferIfTooBig(int threshold) which will reallocate zzBuffer if
> > it
> > > > exceeded the threshold. We can decide on a reasonable 64K threshold
> > or
> > > > something, or simply always cut back to 16 KB. As far as I
> > understand,
> > > that
> > > > buffer should never grow that much. I.e. in zzRefill, which is the
> > only
> > > > place where the buffer gets resized, there is an attempt to first
> > move
> > > back
> > > > characters that were already consumed and only then allocate a
> > bigger
> > > > buffer. Which means only if there is a token whose size is larger
> > than
> > > 16KB
> > > > (!?), will this buffer get expanded.
> > > >
> > > > A trimBuffer method might not be that bad .. as a protective
> > measure.
> > > What
> > > > do you think? Of course, JFlex can fix it on their own ... but
> > until that
> > > > happens ...
> > > >
> > > > Shai
> > > >
> > > > On Thu, Apr 8, 2010 at 10:35 AM, Uwe Schindler <uw...@thetaphi.de>
> > wrote:
> > > >
> > > > > > I would like to identify also the problematic document I have
> > 10000
> > > so,
> > > > > > what
> > > > > > would be the best way of identifying the one that it making
> > zzBuffer
> > > to
> > > > > > grow
> > > > > > without control?
> > > > >
> > > > > Dont index your documents, but instead pass them directly to the
> > > analyzer
> > > > > and consume the tokenstream manually. Then visit
> > > > TermAttribute.termLength()
> > > > > for each Token.
> > > > >
> > > > >
> > > > > -----------------------------------------------------------------
> > ----
> > > > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > > > > For additional commands, e-mail: java-user-help@lucene.apache.org
> > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > /Rubén
> > >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
RE: IndexWriter memory leak?
Posted by Uwe Schindler <uw...@thetaphi.de>.
I already answered, that I will take care of this!
Uwe
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: Shai Erera [mailto:serera@gmail.com]
> Sent: Thursday, April 08, 2010 12:00 PM
> To: java-user@lucene.apache.org
> Subject: Re: IndexWriter memory leak?
>
> Yes, that's the trimBuffer version I was thinking about, only this guy
> created a reset(Reader, int) and does both ops (resetting + trim) in
> one
> method call. More convenient. Can you please open an issue to track
> that?
> People will have a chance to comment on whether we (Lucene) should
> handle
> that, or it should be a JFlex fix. Based on the number of replies this
> guy
> received (0 !), I doubt JFlex would consider it a problem. But we can
> do
> some small service to our users base by protecting against such
> problems.
>
> And while you're opening the issue, if you want to take a stab at
> fixing it
> and post a patch, it'd be great :).
>
> Shai
>
> On Thu, Apr 8, 2010 at 12:51 PM, Ruben Laguna
> <ru...@gmail.com>wrote:
>
> > I was investigating this a little further and in the JFlex mailing
> list I
> > found [1]
> >
> > I don't know much about flex / JFlex but it seems that this guy
> resets the
> > zzBuffer to 16384 or less when setting the input for the lexer
> >
> >
> > Quoted from shef <sh...@ya...>
> >
> >
> > I set
> >
> > %buffer 0
> >
> > in the options section, and then added this method to the lexer:
> >
> > /**
> > * Set the input for the lexer. The size parameter really speeds
> things
> > up,
> > * because by default, the lexer allocates an internal buffer of
> 16k.
> > For
> > * most strings, this is unnecessarily large. If the size param is
> > 0 or greater
> > * than 16k, then the buffer is set to 16k. If the size param is
> > smaller, then
> > * the buf will be set to the exact size.
> > * @param r the reader that provides the data
> > * @param the size of the data in the reader.
> > */
> > public void reset(Reader r, int size) {
> > if (size == 0 || size > 16384)
> > size = 16384;
> > zzBuffer = new char[size];
> > yyreset(r);
> > }
> >
> >
> > So maybe there is a way to trim the zzBuffer this way (?).
> >
> > BTW, I will try to find out which is the "big token" in my dataset
> this
> > afternoon. Thanks for the help.
> >
> > I actually workaround this memory problem for the time being by
> wrapping
> > the
> > IndexWriter in a class that periodically closes the IndexWriter and
> creates
> > a new one, allowing the old to be GCed, but I would be really good if
> > either
> > JFlex or Lucene can take care of this zzBuffer going berserk.
> >
> >
> > Again thanks for the quick response. /Rubén
> >
> >
> > [1]
> >
> >
> https://sourceforge.net/mailarchive/message.php?msg_id=444070.38422.qm@
> web38901.mail.mud.yahoo.com
> >
> > On Thu, Apr 8, 2010 at 11:32 AM, Shai Erera <se...@gmail.com> wrote:
> >
> > > If we could change the Flex file so that yyreset(Reader) would
> check the
> > > size of zzBuffer, we could trim it when it gets too big. But I
> don't
> > think
> > > we have such control when writing the flex syntax ... yyreset is
> > generated
> > > by JFlex and that's the only place I can think of to trim the
> buffer down
> > > when it exceeds a predefined threshold ....
> > >
> > > Maybe what we can do is create our own method which will be called
> by
> > > StandardTokenizer after yyreset is called, something like
> > > trimBufferIfTooBig(int threshold) which will reallocate zzBuffer if
> it
> > > exceeded the threshold. We can decide on a reasonable 64K threshold
> or
> > > something, or simply always cut back to 16 KB. As far as I
> understand,
> > that
> > > buffer should never grow that much. I.e. in zzRefill, which is the
> only
> > > place where the buffer gets resized, there is an attempt to first
> move
> > back
> > > characters that were already consumed and only then allocate a
> bigger
> > > buffer. Which means only if there is a token whose size is larger
> than
> > 16KB
> > > (!?), will this buffer get expanded.
> > >
> > > A trimBuffer method might not be that bad .. as a protective
> measure.
> > What
> > > do you think? Of course, JFlex can fix it on their own ... but
> until that
> > > happens ...
> > >
> > > Shai
> > >
> > > On Thu, Apr 8, 2010 at 10:35 AM, Uwe Schindler <uw...@thetaphi.de>
> wrote:
> > >
> > > > > I would like to identify also the problematic document I have
> 10000
> > so,
> > > > > what
> > > > > would be the best way of identifying the one that it making
> zzBuffer
> > to
> > > > > grow
> > > > > without control?
> > > >
> > > > Dont index your documents, but instead pass them directly to the
> > analyzer
> > > > and consume the tokenstream manually. Then visit
> > > TermAttribute.termLength()
> > > > for each Token.
> > > >
> > > >
> > > > -----------------------------------------------------------------
> ----
> > > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > > > For additional commands, e-mail: java-user-help@lucene.apache.org
> > > >
> > > >
> > >
> >
> >
> >
> > --
> > /Rubén
> >
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: IndexWriter memory leak?
Posted by Shai Erera <se...@gmail.com>.
Yes, that's the trimBuffer version I was thinking about, only this guy
created a reset(Reader, int) and does both ops (resetting + trim) in one
method call. More convenient. Can you please open an issue to track that?
People will have a chance to comment on whether we (Lucene) should handle
that, or it should be a JFlex fix. Based on the number of replies this guy
received (0 !), I doubt JFlex would consider it a problem. But we can do
some small service to our users base by protecting against such problems.
And while you're opening the issue, if you want to take a stab at fixing it
and post a patch, it'd be great :).
Shai
On Thu, Apr 8, 2010 at 12:51 PM, Ruben Laguna <ru...@gmail.com>wrote:
> I was investigating this a little further and in the JFlex mailing list I
> found [1]
>
> I don't know much about flex / JFlex but it seems that this guy resets the
> zzBuffer to 16384 or less when setting the input for the lexer
>
>
> Quoted from shef <sh...@ya...>
>
>
> I set
>
> %buffer 0
>
> in the options section, and then added this method to the lexer:
>
> /**
> * Set the input for the lexer. The size parameter really speeds things
> up,
> * because by default, the lexer allocates an internal buffer of 16k.
> For
> * most strings, this is unnecessarily large. If the size param is
> 0 or greater
> * than 16k, then the buffer is set to 16k. If the size param is
> smaller, then
> * the buf will be set to the exact size.
> * @param r the reader that provides the data
> * @param the size of the data in the reader.
> */
> public void reset(Reader r, int size) {
> if (size == 0 || size > 16384)
> size = 16384;
> zzBuffer = new char[size];
> yyreset(r);
> }
>
>
> So maybe there is a way to trim the zzBuffer this way (?).
>
> BTW, I will try to find out which is the "big token" in my dataset this
> afternoon. Thanks for the help.
>
> I actually workaround this memory problem for the time being by wrapping
> the
> IndexWriter in a class that periodically closes the IndexWriter and creates
> a new one, allowing the old to be GCed, but I would be really good if
> either
> JFlex or Lucene can take care of this zzBuffer going berserk.
>
>
> Again thanks for the quick response. /Rubén
>
>
> [1]
>
> https://sourceforge.net/mailarchive/message.php?msg_id=444070.38422.qm@web38901.mail.mud.yahoo.com
>
> On Thu, Apr 8, 2010 at 11:32 AM, Shai Erera <se...@gmail.com> wrote:
>
> > If we could change the Flex file so that yyreset(Reader) would check the
> > size of zzBuffer, we could trim it when it gets too big. But I don't
> think
> > we have such control when writing the flex syntax ... yyreset is
> generated
> > by JFlex and that's the only place I can think of to trim the buffer down
> > when it exceeds a predefined threshold ....
> >
> > Maybe what we can do is create our own method which will be called by
> > StandardTokenizer after yyreset is called, something like
> > trimBufferIfTooBig(int threshold) which will reallocate zzBuffer if it
> > exceeded the threshold. We can decide on a reasonable 64K threshold or
> > something, or simply always cut back to 16 KB. As far as I understand,
> that
> > buffer should never grow that much. I.e. in zzRefill, which is the only
> > place where the buffer gets resized, there is an attempt to first move
> back
> > characters that were already consumed and only then allocate a bigger
> > buffer. Which means only if there is a token whose size is larger than
> 16KB
> > (!?), will this buffer get expanded.
> >
> > A trimBuffer method might not be that bad .. as a protective measure.
> What
> > do you think? Of course, JFlex can fix it on their own ... but until that
> > happens ...
> >
> > Shai
> >
> > On Thu, Apr 8, 2010 at 10:35 AM, Uwe Schindler <uw...@thetaphi.de> wrote:
> >
> > > > I would like to identify also the problematic document I have 10000
> so,
> > > > what
> > > > would be the best way of identifying the one that it making zzBuffer
> to
> > > > grow
> > > > without control?
> > >
> > > Dont index your documents, but instead pass them directly to the
> analyzer
> > > and consume the tokenstream manually. Then visit
> > TermAttribute.termLength()
> > > for each Token.
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > > For additional commands, e-mail: java-user-help@lucene.apache.org
> > >
> > >
> >
>
>
>
> --
> /Rubén
>
Re: IndexWriter memory leak?
Posted by Ruben Laguna <ru...@gmail.com>.
I was investigating this a little further and in the JFlex mailing list I
found [1]
I don't know much about flex / JFlex but it seems that this guy resets the
zzBuffer to 16384 or less when setting the input for the lexer
Quoted from shef <sh...@ya...>
I set
%buffer 0
in the options section, and then added this method to the lexer:
/**
* Set the input for the lexer. The size parameter really speeds things up,
* because by default, the lexer allocates an internal buffer of 16k. For
* most strings, this is unnecessarily large. If the size param is
0 or greater
* than 16k, then the buffer is set to 16k. If the size param is
smaller, then
* the buf will be set to the exact size.
* @param r the reader that provides the data
* @param the size of the data in the reader.
*/
public void reset(Reader r, int size) {
if (size == 0 || size > 16384)
size = 16384;
zzBuffer = new char[size];
yyreset(r);
}
So maybe there is a way to trim the zzBuffer this way (?).
BTW, I will try to find out which is the "big token" in my dataset this
afternoon. Thanks for the help.
I actually workaround this memory problem for the time being by wrapping the
IndexWriter in a class that periodically closes the IndexWriter and creates
a new one, allowing the old to be GCed, but I would be really good if either
JFlex or Lucene can take care of this zzBuffer going berserk.
Again thanks for the quick response. /Rubén
[1]
https://sourceforge.net/mailarchive/message.php?msg_id=444070.38422.qm@web38901.mail.mud.yahoo.com
On Thu, Apr 8, 2010 at 11:32 AM, Shai Erera <se...@gmail.com> wrote:
> If we could change the Flex file so that yyreset(Reader) would check the
> size of zzBuffer, we could trim it when it gets too big. But I don't think
> we have such control when writing the flex syntax ... yyreset is generated
> by JFlex and that's the only place I can think of to trim the buffer down
> when it exceeds a predefined threshold ....
>
> Maybe what we can do is create our own method which will be called by
> StandardTokenizer after yyreset is called, something like
> trimBufferIfTooBig(int threshold) which will reallocate zzBuffer if it
> exceeded the threshold. We can decide on a reasonable 64K threshold or
> something, or simply always cut back to 16 KB. As far as I understand, that
> buffer should never grow that much. I.e. in zzRefill, which is the only
> place where the buffer gets resized, there is an attempt to first move back
> characters that were already consumed and only then allocate a bigger
> buffer. Which means only if there is a token whose size is larger than 16KB
> (!?), will this buffer get expanded.
>
> A trimBuffer method might not be that bad .. as a protective measure. What
> do you think? Of course, JFlex can fix it on their own ... but until that
> happens ...
>
> Shai
>
> On Thu, Apr 8, 2010 at 10:35 AM, Uwe Schindler <uw...@thetaphi.de> wrote:
>
> > > I would like to identify also the problematic document I have 10000 so,
> > > what
> > > would be the best way of identifying the one that it making zzBuffer to
> > > grow
> > > without control?
> >
> > Dont index your documents, but instead pass them directly to the analyzer
> > and consume the tokenstream manually. Then visit
> TermAttribute.termLength()
> > for each Token.
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
>
--
/Rubén
Re: IndexWriter memory leak?
Posted by Shai Erera <se...@gmail.com>.
If we could change the Flex file so that yyreset(Reader) would check the
size of zzBuffer, we could trim it when it gets too big. But I don't think
we have such control when writing the flex syntax ... yyreset is generated
by JFlex and that's the only place I can think of to trim the buffer down
when it exceeds a predefined threshold ....
Maybe what we can do is create our own method which will be called by
StandardTokenizer after yyreset is called, something like
trimBufferIfTooBig(int threshold) which will reallocate zzBuffer if it
exceeded the threshold. We can decide on a reasonable 64K threshold or
something, or simply always cut back to 16 KB. As far as I understand, that
buffer should never grow that much. I.e. in zzRefill, which is the only
place where the buffer gets resized, there is an attempt to first move back
characters that were already consumed and only then allocate a bigger
buffer. Which means only if there is a token whose size is larger than 16KB
(!?), will this buffer get expanded.
A trimBuffer method might not be that bad .. as a protective measure. What
do you think? Of course, JFlex can fix it on their own ... but until that
happens ...
Shai
On Thu, Apr 8, 2010 at 10:35 AM, Uwe Schindler <uw...@thetaphi.de> wrote:
> > I would like to identify also the problematic document I have 10000 so,
> > what
> > would be the best way of identifying the one that it making zzBuffer to
> > grow
> > without control?
>
> Dont index your documents, but instead pass them directly to the analyzer
> and consume the tokenstream manually. Then visit TermAttribute.termLength()
> for each Token.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
RE: IndexWriter memory leak?
Posted by Uwe Schindler <uw...@thetaphi.de>.
> I would like to identify also the problematic document I have 10000 so,
> what
> would be the best way of identifying the one that it making zzBuffer to
> grow
> without control?
Dont index your documents, but instead pass them directly to the analyzer and consume the tokenstream manually. Then visit TermAttribute.termLength() for each Token.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: IndexWriter memory leak?
Posted by Uwe Schindler <uw...@thetaphi.de>.
Hi Ruben,
as Shai already pointed out, the buffer with this large size is hold by "StandardTokenizer", which is used in the "StandardAnalyzer". This code is out of Lucene's control, as it is generated by the jFlex library.
As long as the IndexWriter instance is living, the buffer is hold implicitely by the Analyzer. If you want to shrink the buffer, file a bug against JFlex. For reuse, StandardAnalyzer holds the Tokenizer instance as long as it lives. As you pass the Analyzer to the IndexWriter it will only be GCed when IndexWriter is GCed.
Uwe
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: Ruben Laguna [mailto:ruben.laguna@gmail.com]
> Sent: Thursday, April 08, 2010 9:24 AM
> To: java-user@lucene.apache.org
> Subject: Re: IndexWriter memory leak?
>
> I'm using StandardAnalyzer.
>
> I indeed parse large documents, xml and pdfs, using nekohtml and tika
> respectively.
>
> I took a look to the zzBuffer value contents (by exporting it to a file
> with
> Eclipse MAT from the heapdump) and it seems to contain normal text from
> several documents. See below
>
> cat heapdumps/zzBuffer/value|iconv -f UTF-16 -t UTF-8|head
>
> On the NetBeans Platform Build Harness
> «/on-the-netbeans-platform-build-harness/atform Build Harness
>
> Abstract
> The NetBeans build harness performs a number of tasks when building a
> NetBeans platform application. This article describes some of the
> issues
> that can be encountered when building platform applications in other
> ways
> than the standard use case of creating and building it on a single PC
> in the
> NetBeans IDE. It also provides some tips for customising build
> behaviour.
>
>
> but it's difficult to tell if there is also a big token in the file due
> to
> the fact that I have no easy way to visualize it. (iconv is skipping
> things
> ). (hexdump -C doesn't really help to see the contents).
>
> Do you know of anyway of actually indentifying the offending token or
> to
> list the tokens in the zzBuffer in an human readable way?
>
> I would like to identify also the problematic document I have 10000 so,
> what
> would be the best way of identifying the one that it making zzBuffer to
> grow
> without control?
>
> Best regards/Ruben
>
> On Thu, Apr 8, 2010 at 8:41 AM, Shai Erera <se...@gmail.com> wrote:
>
> > What Analyzer are you using? zzBuffer belongs to the tokenizer's
> automaton
> > that is generated by JFlex. I've checked StandardTokenizerImpl and
> zzBuffer
> > can grow, beyond the default 16KB, but yours look to be a lot bigger
> (33 MB
> > !?). The only explanation I have to this is that you're trying to (or
> in
> > the
> > process have tried to) parse a very long file, which its text
> couldn't be
> > broken down to small tokens (maybe due to a lack of spaces or
> something)
> > ...
> >
> > It'd be good if you tell us the Analyzer you use as well as if you
> can
> > identify a problematic document in your collection which can cause
> the
> > buffer to grow that much.
> >
> > Shai
> >
> > On Thu, Apr 8, 2010 at 1:23 AM, Ruben Laguna <ru...@gmail.com>
> > wrote:
> >
> > > I want to add that it tried this in both 2.9.0 and 3.0.1 and I got
> the
> > same
> > > "leaky" behavior.
> > >
> > > See [3] for a screenshot of zzBuffer of 67MB on Lucene 3.0.1.
> > >
> > > I managed to get rid of the Reader's "memory leak" by actually
> setting to
> > > null the pointer to the actual Tika's Reader in my wrapper when the
> > wrapper
> > > is closed. But I still think that I would be nicer if IndexWriter
> > wouldn't
> > > maintain references to the Readers after indexing.
> > >
> > > [3] http://img.skitch.com/20100407-ntn2kg13fx49wx4q118bp9h1hb.jpg
> > >
> > >
> > > On Wed, Apr 7, 2010 at 10:35 PM, Ruben Laguna
> <ruben.laguna@gmail.com
> > > >wrote:
> > >
> > > > Hi,
> > > >
> > > > It seems like my IndexWriter after commiting and optimizing has a
> > > retained
> > > > size of 140Mb. See [1] for a screenshot of the heapdump analysis
> done
> > > with
> > > > Eclipse MAT.
> > > >
> > > > Of those 140MB 67MB are retained by
> > > >
> > >
> >
> analyzer.tokenStreams.hardRefs.table.HashMap$Entry.value.tokenStream.sc
> anner.zzBuffer
> > > >
> > > >
> > > > why is this? Is it a memory leak? or did I something wrong during
> the
> > > > indxing? (BTW, I'm indexing document which contains
> Fields(xxxx,Reader)
> > > and
> > > > those Reader are wrappers around Tika.parse(xxxx) Readers. I get
> a lot
> > > > IOExceptions from tika readers and the wrapper maps the
> exceptions to
> > EOF
> > > so
> > > > Lucene doesn't see the exception).
> > > >
> > > >
> > > >
> > > > ...and 73MB of the 140MB are retained by docWriter see [2]. It
> looks
> > like
> > > > the Field objects in the
> > > > array
> docWriter.threadStates[0].consumer.fieldHash[1].fields[xxxx] are
> > > > holding references to the Readers. Those reader instances are
> actually
> > > > closed after IndexWriter.updateDocument. Each one of those
> Readers
> > > retains
> > > > 1MB. The question is why IndexWriter holds references to those
> Readers
> > > after
> > > > the Documents have been indexed.
> > > >
> > > >
> > > > [1] http://img.skitch.com/20100407-1183815yiausisg73u9wfgscsj.jpg
> > > > [2] http://img.skitch.com/20100407-b86irkp7e4uif2wq1dd4t899qb.jpg
> > > >
> > > > --
> > > > /Rubén
> > > >
> > >
> > >
> > >
> > > --
> > > /Rubén
> > >
> >
>
>
>
> --
> /Rubén
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: IndexWriter memory leak?
Posted by Ruben Laguna <ru...@gmail.com>.
I'm using StandardAnalyzer.
I indeed parse large documents, xml and pdfs, using nekohtml and tika
respectively.
I took a look to the zzBuffer value contents (by exporting it to a file with
Eclipse MAT from the heapdump) and it seems to contain normal text from
several documents. See below
cat heapdumps/zzBuffer/value|iconv -f UTF-16 -t UTF-8|head
On the NetBeans Platform Build Harness
«/on-the-netbeans-platform-build-harness/atform Build Harness
Abstract
The NetBeans build harness performs a number of tasks when building a
NetBeans platform application. This article describes some of the issues
that can be encountered when building platform applications in other ways
than the standard use case of creating and building it on a single PC in the
NetBeans IDE. It also provides some tips for customising build behaviour.
but it's difficult to tell if there is also a big token in the file due to
the fact that I have no easy way to visualize it. (iconv is skipping things
). (hexdump -C doesn't really help to see the contents).
Do you know of anyway of actually indentifying the offending token or to
list the tokens in the zzBuffer in an human readable way?
I would like to identify also the problematic document I have 10000 so, what
would be the best way of identifying the one that it making zzBuffer to grow
without control?
Best regards/Ruben
On Thu, Apr 8, 2010 at 8:41 AM, Shai Erera <se...@gmail.com> wrote:
> What Analyzer are you using? zzBuffer belongs to the tokenizer's automaton
> that is generated by JFlex. I've checked StandardTokenizerImpl and zzBuffer
> can grow, beyond the default 16KB, but yours look to be a lot bigger (33 MB
> !?). The only explanation I have to this is that you're trying to (or in
> the
> process have tried to) parse a very long file, which its text couldn't be
> broken down to small tokens (maybe due to a lack of spaces or something)
> ...
>
> It'd be good if you tell us the Analyzer you use as well as if you can
> identify a problematic document in your collection which can cause the
> buffer to grow that much.
>
> Shai
>
> On Thu, Apr 8, 2010 at 1:23 AM, Ruben Laguna <ru...@gmail.com>
> wrote:
>
> > I want to add that it tried this in both 2.9.0 and 3.0.1 and I got the
> same
> > "leaky" behavior.
> >
> > See [3] for a screenshot of zzBuffer of 67MB on Lucene 3.0.1.
> >
> > I managed to get rid of the Reader's "memory leak" by actually setting to
> > null the pointer to the actual Tika's Reader in my wrapper when the
> wrapper
> > is closed. But I still think that I would be nicer if IndexWriter
> wouldn't
> > maintain references to the Readers after indexing.
> >
> > [3] http://img.skitch.com/20100407-ntn2kg13fx49wx4q118bp9h1hb.jpg
> >
> >
> > On Wed, Apr 7, 2010 at 10:35 PM, Ruben Laguna <ruben.laguna@gmail.com
> > >wrote:
> >
> > > Hi,
> > >
> > > It seems like my IndexWriter after commiting and optimizing has a
> > retained
> > > size of 140Mb. See [1] for a screenshot of the heapdump analysis done
> > with
> > > Eclipse MAT.
> > >
> > > Of those 140MB 67MB are retained by
> > >
> >
> analyzer.tokenStreams.hardRefs.table.HashMap$Entry.value.tokenStream.scanner.zzBuffer
> > >
> > >
> > > why is this? Is it a memory leak? or did I something wrong during the
> > > indxing? (BTW, I'm indexing document which contains Fields(xxxx,Reader)
> > and
> > > those Reader are wrappers around Tika.parse(xxxx) Readers. I get a lot
> > > IOExceptions from tika readers and the wrapper maps the exceptions to
> EOF
> > so
> > > Lucene doesn't see the exception).
> > >
> > >
> > >
> > > ...and 73MB of the 140MB are retained by docWriter see [2]. It looks
> like
> > > the Field objects in the
> > > array docWriter.threadStates[0].consumer.fieldHash[1].fields[xxxx] are
> > > holding references to the Readers. Those reader instances are actually
> > > closed after IndexWriter.updateDocument. Each one of those Readers
> > retains
> > > 1MB. The question is why IndexWriter holds references to those Readers
> > after
> > > the Documents have been indexed.
> > >
> > >
> > > [1] http://img.skitch.com/20100407-1183815yiausisg73u9wfgscsj.jpg
> > > [2] http://img.skitch.com/20100407-b86irkp7e4uif2wq1dd4t899qb.jpg
> > >
> > > --
> > > /Rubén
> > >
> >
> >
> >
> > --
> > /Rubén
> >
>
--
/Rubén
Re: IndexWriter memory leak?
Posted by Shai Erera <se...@gmail.com>.
What Analyzer are you using? zzBuffer belongs to the tokenizer's automaton
that is generated by JFlex. I've checked StandardTokenizerImpl and zzBuffer
can grow, beyond the default 16KB, but yours look to be a lot bigger (33 MB
!?). The only explanation I have to this is that you're trying to (or in the
process have tried to) parse a very long file, which its text couldn't be
broken down to small tokens (maybe due to a lack of spaces or something) ...
It'd be good if you tell us the Analyzer you use as well as if you can
identify a problematic document in your collection which can cause the
buffer to grow that much.
Shai
On Thu, Apr 8, 2010 at 1:23 AM, Ruben Laguna <ru...@gmail.com> wrote:
> I want to add that it tried this in both 2.9.0 and 3.0.1 and I got the same
> "leaky" behavior.
>
> See [3] for a screenshot of zzBuffer of 67MB on Lucene 3.0.1.
>
> I managed to get rid of the Reader's "memory leak" by actually setting to
> null the pointer to the actual Tika's Reader in my wrapper when the wrapper
> is closed. But I still think that I would be nicer if IndexWriter wouldn't
> maintain references to the Readers after indexing.
>
> [3] http://img.skitch.com/20100407-ntn2kg13fx49wx4q118bp9h1hb.jpg
>
>
> On Wed, Apr 7, 2010 at 10:35 PM, Ruben Laguna <ruben.laguna@gmail.com
> >wrote:
>
> > Hi,
> >
> > It seems like my IndexWriter after commiting and optimizing has a
> retained
> > size of 140Mb. See [1] for a screenshot of the heapdump analysis done
> with
> > Eclipse MAT.
> >
> > Of those 140MB 67MB are retained by
> >
> analyzer.tokenStreams.hardRefs.table.HashMap$Entry.value.tokenStream.scanner.zzBuffer
> >
> >
> > why is this? Is it a memory leak? or did I something wrong during the
> > indxing? (BTW, I'm indexing document which contains Fields(xxxx,Reader)
> and
> > those Reader are wrappers around Tika.parse(xxxx) Readers. I get a lot
> > IOExceptions from tika readers and the wrapper maps the exceptions to EOF
> so
> > Lucene doesn't see the exception).
> >
> >
> >
> > ...and 73MB of the 140MB are retained by docWriter see [2]. It looks like
> > the Field objects in the
> > array docWriter.threadStates[0].consumer.fieldHash[1].fields[xxxx] are
> > holding references to the Readers. Those reader instances are actually
> > closed after IndexWriter.updateDocument. Each one of those Readers
> retains
> > 1MB. The question is why IndexWriter holds references to those Readers
> after
> > the Documents have been indexed.
> >
> >
> > [1] http://img.skitch.com/20100407-1183815yiausisg73u9wfgscsj.jpg
> > [2] http://img.skitch.com/20100407-b86irkp7e4uif2wq1dd4t899qb.jpg
> >
> > --
> > /Rubén
> >
>
>
>
> --
> /Rubén
>
Re: IndexWriter memory leak?
Posted by Ruben Laguna <ru...@gmail.com>.
I want to add that it tried this in both 2.9.0 and 3.0.1 and I got the same
"leaky" behavior.
See [3] for a screenshot of zzBuffer of 67MB on Lucene 3.0.1.
I managed to get rid of the Reader's "memory leak" by actually setting to
null the pointer to the actual Tika's Reader in my wrapper when the wrapper
is closed. But I still think that I would be nicer if IndexWriter wouldn't
maintain references to the Readers after indexing.
[3] http://img.skitch.com/20100407-ntn2kg13fx49wx4q118bp9h1hb.jpg
On Wed, Apr 7, 2010 at 10:35 PM, Ruben Laguna <ru...@gmail.com>wrote:
> Hi,
>
> It seems like my IndexWriter after commiting and optimizing has a retained
> size of 140Mb. See [1] for a screenshot of the heapdump analysis done with
> Eclipse MAT.
>
> Of those 140MB 67MB are retained by
> analyzer.tokenStreams.hardRefs.table.HashMap$Entry.value.tokenStream.scanner.zzBuffer
>
>
> why is this? Is it a memory leak? or did I something wrong during the
> indxing? (BTW, I'm indexing document which contains Fields(xxxx,Reader) and
> those Reader are wrappers around Tika.parse(xxxx) Readers. I get a lot
> IOExceptions from tika readers and the wrapper maps the exceptions to EOF so
> Lucene doesn't see the exception).
>
>
>
> ...and 73MB of the 140MB are retained by docWriter see [2]. It looks like
> the Field objects in the
> array docWriter.threadStates[0].consumer.fieldHash[1].fields[xxxx] are
> holding references to the Readers. Those reader instances are actually
> closed after IndexWriter.updateDocument. Each one of those Readers retains
> 1MB. The question is why IndexWriter holds references to those Readers after
> the Documents have been indexed.
>
>
> [1] http://img.skitch.com/20100407-1183815yiausisg73u9wfgscsj.jpg
> [2] http://img.skitch.com/20100407-b86irkp7e4uif2wq1dd4t899qb.jpg
>
> --
> /Rubén
>
--
/Rubén