You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Paul Allan Hill <pa...@metajure.com> on 2012/01/28 00:18:57 UTC
deprecated optimize()!
After reading all about the renaming of optimize() and updating my Lucene libraries to 3.4, I was surprised and confused by what I found.
I have a 1 segment index (all files are named _1*.*) that had been created with 3.0.1 code which had been optimized many times (all 3.0.1 code). The first time my code used the 3.4 libraries with version level set to 3.4 and it tried to optimize() (still using this now deprecated old call), the new code went wild! It took up more memory than the heap was limited to, so I believe it is taking up system resources. We have turned off optimize for now.
What is it doing? (I believe it is using a TieredMergePolicy, see the IndexWriter constructor).
Anyone have any resources to point me to on the subject so I can pick the right call to forceMerge() and a good MergePolicy before upgrading to Lucene 4.0?
Any ideas? What should be a reasonable 3.4 approach? Are there issues about the the just complete IndexWriter? Ho about open readers (in the web app using the index).
-Paul
p.s. More details available upon request.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: deprecated optimize()!
Posted by Uwe Schindler <uw...@thetaphi.de>.
Hi,
> > > The first time my code used the 3.4 libraries with version level set
> > > to 3.4 and it tried to optimize() (still using this now deprecated
> > > old call), the new code
> > went wild!
> > > It took up more memory than the heap was limited to, so I believe it
> > > is taking
> > > up system resources. We have turned off optimize for now.
> >
> > Did it throw OutOfMemoryException?
>
> As you thought, there was No OOM Exception.
>
> >I assume not - but I assume, you have seen more virtual memory usage
> >in "top", but that's not really related to optimize/forceMerge.
>
> I'm actually on a large Windows server box.
On Windows x64, Lucene will also automatically use memory mapping, too. What
you see in Windows Task Manager is also virtual memory usage. If the virtual
memory usage is much bigger than the actual heap -Xmx allocated that exactly
coming from memory-mapped files.
> It is only related to optimize() in that all memory is used _only_ when
the
> server goes for an optimize().
> Running through our web interface (which reuses its one IndexReader)
doesn't
> seem to drive up the memory (beyond use of the heap).
If you optimize, the system is having the index files multiple times mapped
into virtual address space. This is not heap space used (otherwise your
system would OOM).
> The problem is that when we run the optimize, the machine gets memory
> limited and everything then runs slow (nearly preventing us from viewing
what
> going on) and the optimize() takes "forever" (=an extra 1.5 hrs). We are
not
> trying to share the machine, but it is a bit excessive.
How big is you index and how big are system memory and how much is allocated
to the JVM heap? Please note: Never allocate too much memory to the JVM,
most parts of Lucene are running in file system cache (like the
memory-mapped files), if you allocate too much -Xmx heap, the system will
hit millions of page faults and the kernel needs to swap in/out the index
files all the time. This affects optimize, as this transfers lots of data
causing page-faults all over, stealing also mapped pages from your active
IndexReader serving the searches. Depending on some factors (like sorting),
you should only allocate very little heap (by testing and waiting for OOM)
and keep the rest of the physical memory available for the FS cache.
To enable the Mmapping of index files in Lucene 3.0.1, you can use "new
MMapDirectory()" instead "FSDirectory.open()", then 3.0.1 will have the
similar virtual memory usage like 3.4. You can also force Lucene 3.4 to the
slower SimpleFSDirectory by instantiating that directly (which emulates
Lucene 3.0.1 file access on Windows). But this will slow down your searches!
> Further reading suggests that maybe the right approach is to wait for
> IndexWriter to insert cleanup as the need arises. Does that sound right?
Exactly, simple answer: don't optimize! But you should still review your
heap paremeters, as there seems to be something very misconfigured in your
system (see above).
> Is my read that the system will prevent excessive segments and cleanup
> deleted documents on its own a reasonable assessment of state of Lucene in
> 3.4 or 4.0? Thus I don't really need to call optimize()/maybeMerge().
> If I skip optimize()/ maybeMerge() am I missing anything I should be doing
to
> keep the index 'tidy'? What I don't want is to find the index runs slow
only after
> months of many document updates.
A "tidy" index is not necessarily one with one segment and no deletions.
That's just a "compact" representation, but not always the fastest. The
slowdown by forceMerge(1) is much worse than the little overhead by
deletions and multiple segments. If you use the new IndexSearcher ctors with
ExecutorService, an unoptimized index may even be faster on searching, as it
can handle the segment searches in different threads!
The slowdowns of older Lucene versions with unoptimized indexes are solved
since 2.9, because searches are running on every segment privately and no
longer on a "MultiReader"-like approach that needs to merge terms and
postings on the fly (which causes slowness in pre-2.9).
More info about not using optimizing here: http://goo.gl/Dl1jO
> Any discussion by those with similar indexes would be welcome.
>
> -Paul
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: deprecated optimize()!
Posted by Paul Allan Hill <pa...@metajure.com>.
Thanks for the reply,
> > The first time my code used the 3.4 libraries with version level set
> > to 3.4 and it tried
> > to optimize() (still using this now deprecated old call), the new code
> went wild!
> > It took up more memory than the heap was limited to, so I believe it
> > is taking
> > up system resources. We have turned off optimize for now.
>
> Did it throw OutOfMemoryException?
As you thought, there was No OOM Exception.
>I assume not - but I assume, you have seen more virtual
> memory usage in "top", but that's not really related to optimize/forceMerge.
I'm actually on a large Windows server box.
It is only related to optimize() in that all memory is used _only_ when the server goes for an optimize().
Running through our web interface (which reuses its one IndexReader) doesn't seem to drive up the memory (beyond use of the heap).
The problem is that when we run the optimize, the machine gets memory limited and everything then runs slow (nearly preventing us from viewing what going on) and the optimize() takes "forever" (=an extra 1.5 hrs). We are not trying to share the machine, but it is a bit excessive.
Further reading suggests that maybe the right approach is to wait for IndexWriter to insert cleanup as the need arises. Does that sound right?
Is my read that the system will prevent excessive segments and cleanup deleted documents on its own a reasonable assessment of state of Lucene in 3.4 or 4.0? Thus I don't really need to call optimize()/maybeMerge().
If I skip optimize()/ maybeMerge() am I missing anything I should be doing to keep the index 'tidy'? What I don't want is to find the index runs slow only after months of many document updates.
Any discussion by those with similar indexes would be welcome.
-Paul
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: deprecated optimize()!
Posted by Uwe Schindler <uw...@thetaphi.de>.
Hi,
> After reading all about the renaming of optimize() and updating my Lucene
> libraries to 3.4, I was surprised and confused by what I found.
>
> I have a 1 segment index (all files are named _1*.*) that had been
created
> with 3.0.1 code which had been optimized many times (all 3.0.1 code). The
> first time my code used the 3.4 libraries with version level set to 3.4
and it tried
> to optimize() (still using this now deprecated old call), the new code
went wild!
> It took up more memory than the heap was limited to, so I believe it is
taking
> up system resources. We have turned off optimize for now.
Did it throw OutOfMemoryException? I assume not - but I assume, you have
seen more virtual memory usage in "top", but that's not really related to
optimize/forceMerge. If you are running on a 64 bit platform, Lucene 3.4
will default to MMapDirectory on FSDirectory.open(), which maps the index
like a linux/windows swap file into *virtual* memory (not necessary physical
memory, but it then directly accesses the file system cache instead of
copying to Java heap). This is why "top" mem usage grows, even multiple
times of java heap. Nothing to worry about - that's expected and normal (see
http://en.wikipedia.org/wiki/Mmap)!
> What is it doing? (I believe it is using a TieredMergePolicy, see the
IndexWriter
> constructor).
The merge policy does not have anything to do with that, optimizing or
forcemerge to one index segment is the same in all merge policies.
> Anyone have any resources to point me to on the subject so I can pick the
right
> call to forceMerge() and a good MergePolicy before upgrading to Lucene
4.0?
> Any ideas? What should be a reasonable 3.4 approach? Are there issues
> about the the just complete IndexWriter? Ho about open readers (in the
web
> app using the index).
If you have open readers, many files are open and mapped into virtual
memory, so virtual memory usage grows.
> -Paul
>
> p.s. More details available upon request.
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org