Switching existing solr indexes from Segment to Compound Style index files

[Nicholas W <44...@log1.net>]

Dear All,
 On a Linux system running a multi-core linux server, we are
experiencing a problem of too many files open which is causing tomcat
to abort. Reading the documentation, one of the things it seems we can
do is to switch to using compound indexes. We can see that in the
solrconfig.xml there is an option:


   <useCompoundFile>true</useCompoundFile>

in the <indexdefaults> and <mainindex> section. We have set this to
true and restarted tomcat.

I have then used the script ./optimize script to get Solr to optimize
the index. In the lucene documentation it suggests this is the way to
switch to a compound index. However, with SOLR while the index is
optimized, its not converted to a compound file.

What are we doing wrong? What is the correct way to convert an index
to use a compound file?

Thanks a lot for your suggestions.

Regards,
Nicholas

Re: Switching existing solr indexes from Segment to Compound Style index files

[Markus Jelsma <ma...@openindex.io>]

Indeed, wouldn't reducing the number of segments be a better idea? Speeds up 
searching too! Do you happen to have a very high mergeFactor value for each 
core?

On Wednesday 19 January 2011 17:53:12 Erick Erickson wrote:
> You're perhaps exactly right in your approach, but with a bit more info
> we may be able to suggest other alternatives.

-- 
Markus Jelsma - CTO - Openindex
http://www.linkedin.com/in/markus17
050-8536620 / 06-50258350

Re: Switching existing solr indexes from Segment to Compound Style index files

[Erick Erickson <er...@gmail.com>]

Let's back up a ways here and figure out why you're getting so many
files open.

1> how many files are in your index?
2> are you committing very frequently?
3> or do you simply have a LOT of cores?
4> do you optimize your indexes? If so, how many files to you have in your
cores before/after optimizing?

You're perhaps exactly right in your approach, but with a bit more info
we may be able to suggest other alternatives.

Best
Erick

On Wed, Jan 19, 2011 at 4:48 AM, Nicholas W <44...@log1.net> wrote:

> Dear All,
>  On a Linux system running a multi-core linux server, we are
> experiencing a problem of too many files open which is causing tomcat
> to abort. Reading the documentation, one of the things it seems we can
> do is to switch to using compound indexes. We can see that in the
> solrconfig.xml there is an option:
>
>
>   <useCompoundFile>true</useCompoundFile>
>
> in the <indexdefaults> and <mainindex> section. We have set this to
> true and restarted tomcat.
>
> I have then used the script ./optimize script to get Solr to optimize
> the index. In the lucene documentation it suggests this is the way to
> switch to a compound index. However, with SOLR while the index is
> optimized, its not converted to a compound file.
>
> What are we doing wrong? What is the correct way to convert an index
> to use a compound file?
>
> Thanks a lot for your suggestions.
>
> Regards,
> Nicholas
>