You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Qingdi <li...@yahoo.com> on 2009/02/12 00:23:29 UTC
optimization problem
(I posted this question to "solr user" forum, but didn't get a clear answer.
So re-post it here.)
Our index size is about 60G. Most of the time, the optimization works fine.
But this morning, the optimization kept creating new segment files until all
the free disk space (300G) was used up.
Here is how the files generated during optimization look like:
============================================
....................................
-rw-r--r-- 1 root root 213464 Feb 10 07:33 _5w9a.tis
-rw-r--r-- 1 root root 3039 Feb 10 07:33 _5w9a.tii
-rw-r--r-- 1 root root 9360 Feb 10 07:33 _5w9a.prx
-rw-r--r-- 1 root root 19824 Feb 10 07:33 _5w9a.frq
-rw-r--r-- 1 root root 27 Feb 10 07:33 _5w9a.fnm
-rw-r--r-- 1 root root 74884 Feb 10 07:33 _5w9a.fdx
-rw-r--r-- 1 root root 3753944 Feb 10 07:33 _5w9a.fdt
-rw-r--r-- 1 root root 212513 Feb 10 07:33 _5w99.tis
-rw-r--r-- 1 root root 3005 Feb 10 07:33 _5w99.tii
-rw-r--r-- 1 root root 9436 Feb 10 07:33 _5w99.prx
-rw-r--r-- 1 root root 20052 Feb 10 07:33 _5w99.frq
-rw-r--r-- 1 root root 27 Feb 10 07:33 _5w99.fnm
-rw-r--r-- 1 root root 75492 Feb 10 07:33 _5w99.fdx
-rw-r--r-- 1 root root 3795439 Feb 10 07:33 _5w99.fdt
-rw-r--r-- 1 root root 215726 Feb 10 07:33 _5w98.tis
-rw-r--r-- 1 root root 2994 Feb 10 07:33 _5w98.tii
-rw-r--r-- 1 root root 9438 Feb 10 07:33 _5w98.prx
-rw-r--r-- 1 root root 20058 Feb 10 07:33 _5w98.frq
-rw-r--r-- 1 root root 27 Feb 10 07:33 _5w98.fnm
-rw-r--r-- 1 root root 75508 Feb 10 07:33 _5w98.fdx
-rw-r--r-- 1 root root 3779402 Feb 10 07:33 _5w98.fdt
....................................
============================================
This is the third time we encountered this issue. Any idea why the
optimization keeps creating new files? (If we restart the solr server and
run optimization again, the index files can be recovered. we are on solr
1.3.) Where could we find more info. about how the optimization process
work?
Appreciate your help.
Qingdi
--
View this message in context: http://www.nabble.com/optimization-problem-tp21966759p21966759.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: optimization problem
Posted by Michael McCandless <lu...@mikemccandless.com>.
It's odd that optimize is creating such tiny segments, and then that
these tiny segments wind up consuming so much disk space.
Can you turn on IndexWriter's infoStream, and post the output of the
attempts to optimize?
Are you sure there are no unhandled exceptions being logged to the
system error log? By default, merges are done in BG threads (using
ConcurrentMergeScheduler), so exceptions hit in those threads may be
logged elsewhere (though optimize() tries to bring such exceptions to
the foreground thread).
Mike
Qingdi wrote:
>
> (I posted this question to "solr user" forum, but didn't get a clear
> answer.
> So re-post it here.)
>
> Our index size is about 60G. Most of the time, the optimization
> works fine.
> But this morning, the optimization kept creating new segment files
> until all
> the free disk space (300G) was used up.
>
> Here is how the files generated during optimization look like:
> ============================================
> ....................................
> -rw-r--r-- 1 root root 213464 Feb 10 07:33 _5w9a.tis
> -rw-r--r-- 1 root root 3039 Feb 10 07:33 _5w9a.tii
> -rw-r--r-- 1 root root 9360 Feb 10 07:33 _5w9a.prx
> -rw-r--r-- 1 root root 19824 Feb 10 07:33 _5w9a.frq
> -rw-r--r-- 1 root root 27 Feb 10 07:33 _5w9a.fnm
> -rw-r--r-- 1 root root 74884 Feb 10 07:33 _5w9a.fdx
> -rw-r--r-- 1 root root 3753944 Feb 10 07:33 _5w9a.fdt
> -rw-r--r-- 1 root root 212513 Feb 10 07:33 _5w99.tis
> -rw-r--r-- 1 root root 3005 Feb 10 07:33 _5w99.tii
> -rw-r--r-- 1 root root 9436 Feb 10 07:33 _5w99.prx
> -rw-r--r-- 1 root root 20052 Feb 10 07:33 _5w99.frq
> -rw-r--r-- 1 root root 27 Feb 10 07:33 _5w99.fnm
> -rw-r--r-- 1 root root 75492 Feb 10 07:33 _5w99.fdx
> -rw-r--r-- 1 root root 3795439 Feb 10 07:33 _5w99.fdt
> -rw-r--r-- 1 root root 215726 Feb 10 07:33 _5w98.tis
> -rw-r--r-- 1 root root 2994 Feb 10 07:33 _5w98.tii
> -rw-r--r-- 1 root root 9438 Feb 10 07:33 _5w98.prx
> -rw-r--r-- 1 root root 20058 Feb 10 07:33 _5w98.frq
> -rw-r--r-- 1 root root 27 Feb 10 07:33 _5w98.fnm
> -rw-r--r-- 1 root root 75508 Feb 10 07:33 _5w98.fdx
> -rw-r--r-- 1 root root 3779402 Feb 10 07:33 _5w98.fdt
> ....................................
> ============================================
>
> This is the third time we encountered this issue. Any idea why the
> optimization keeps creating new files? (If we restart the solr
> server and
> run optimization again, the index files can be recovered. we are on
> solr
> 1.3.) Where could we find more info. about how the optimization
> process
> work?
>
> Appreciate your help.
>
> Qingdi
> --
> View this message in context: http://www.nabble.com/optimization-problem-tp21966759p21966759.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org