You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by olivier sallou <ol...@gmail.com> on 2011/08/18 17:04:11 UTC
lucene 3 and merge/optimize
Hi,
after an upgrade to solr/lucene 3, I tried to change the code to remove
deprecated functions .... Though new MergePolicy etc... are not really
clear.
I have now issues with the merge and optimize functions.
I have a command line application (Java/Lucene api) that merge multiple
indexes in a single one, or optimize an existing index (this is done
offline)
When I execute my code, the merge creates a new index, but looks to contain
more files than before (with solr 4.1), why not...
When I try to optimize, code says OK, but I still have many files, segments
: (below for a very small example)
_0.fdt _0.tis _1.tii _2.prx _3.nrm _4.frq _5.fnm _6.fdx _7.fdt
_7.tis _8.tii _9.prx _a.nrm _b.frq
_0.fdx _1.fdt _1.tis _2.tii _3.prx _4.nrm _5.frq _6.fnm _7.fdx
_8.fdt _8.tis _9.tii _a.prx _b.nrm
_0.fnm _1.fdx _2.fdt _2.tis _3.tii _4.prx _5.nrm _6.frq _7.fnm
_8.fdx _9.fdt _9.tis _a.tii _b.prx
_0.frq _1.fnm _2.fdx _3.fdt _3.tis _4.tii _5.prx _6.nrm _7.frq
_8.fnm _9.fdx _a.fdt _a.tis _b.tii
_0.nrm _1.frq _2.fnm _3.fdx _4.fdt _4.tis _5.tii _6.prx _7.nrm
_8.frq _9.fnm _a.fdx _b.fdt _b.tis
_0.prx _1.nrm _2.frq _3.fnm _4.fdx _5.fdt _5.tis _6.tii _7.prx
_8.nrm _9.frq _a.fnm _b.fdx segments_1
_0.tii _1.prx _2.nrm _3.frq _4.fnm _5.fdx _6.fdt _6.tis _7.tii
_8.prx _9.nrm _a.frq _b.fnm segments.gen
I'd like to reduce with the optimize or the merge to the minimum the number
of files, my index is read only and does not change.
Here is the code for optimize, am I doing something wrong?
IndexWriterConfig conf = new
IndexWriterConfig(Version.LUCENE_33,newStandardAnalyzer(Version.
LUCENE_33));
conf.setRAMBufferSizeMB(50);
LogByteSizeMergePolicy policy = new LogByteSizeMergePolicy();
policy.setMaxMergeDocs(10);
conf.setMergePolicy(policy);
IndexWriter writer =
newIndexWriter(FSDirectory.open(INDEX_DIR),getIndexConfig() );
writer.optimize();
writer.close();
Thanks
Olivier
Re: lucene 3 and merge/optimize
Posted by olivier sallou <ol...@gmail.com>.
answer to myself, to be checked...
I used policy.setMaxMergeDocs(10), limiting to small number of files....at
least for merge.
I gonna test.
2011/8/18 olivier sallou <ol...@gmail.com>
> Hi,
> after an upgrade to solr/lucene 3, I tried to change the code to remove
> deprecated functions .... Though new MergePolicy etc... are not really
> clear.
>
> I have now issues with the merge and optimize functions.
>
> I have a command line application (Java/Lucene api) that merge multiple
> indexes in a single one, or optimize an existing index (this is done
> offline)
>
> When I execute my code, the merge creates a new index, but looks to contain
> more files than before (with solr 4.1), why not...
> When I try to optimize, code says OK, but I still have many files, segments
> : (below for a very small example)
> _0.fdt _0.tis _1.tii _2.prx _3.nrm _4.frq _5.fnm _6.fdx _7.fdt
> _7.tis _8.tii _9.prx _a.nrm _b.frq
> _0.fdx _1.fdt _1.tis _2.tii _3.prx _4.nrm _5.frq _6.fnm _7.fdx
> _8.fdt _8.tis _9.tii _a.prx _b.nrm
> _0.fnm _1.fdx _2.fdt _2.tis _3.tii _4.prx _5.nrm _6.frq _7.fnm
> _8.fdx _9.fdt _9.tis _a.tii _b.prx
> _0.frq _1.fnm _2.fdx _3.fdt _3.tis _4.tii _5.prx _6.nrm _7.frq
> _8.fnm _9.fdx _a.fdt _a.tis _b.tii
> _0.nrm _1.frq _2.fnm _3.fdx _4.fdt _4.tis _5.tii _6.prx _7.nrm
> _8.frq _9.fnm _a.fdx _b.fdt _b.tis
> _0.prx _1.nrm _2.frq _3.fnm _4.fdx _5.fdt _5.tis _6.tii _7.prx
> _8.nrm _9.frq _a.fnm _b.fdx segments_1
> _0.tii _1.prx _2.nrm _3.frq _4.fnm _5.fdx _6.fdt _6.tis _7.tii
> _8.prx _9.nrm _a.frq _b.fnm segments.gen
>
> I'd like to reduce with the optimize or the merge to the minimum the number
> of files, my index is read only and does not change.
>
> Here is the code for optimize, am I doing something wrong?
>
> IndexWriterConfig conf = new IndexWriterConfig(Version.LUCENE_33,newStandardAnalyzer(Version.
> LUCENE_33));
>
> conf.setRAMBufferSizeMB(50);
>
> LogByteSizeMergePolicy policy = new LogByteSizeMergePolicy();
>
> policy.setMaxMergeDocs(10);
>
> conf.setMergePolicy(policy);
>
> IndexWriter writer = newIndexWriter(FSDirectory.open(INDEX_DIR),getIndexConfig() );
>
>
> writer.optimize();
>
> writer.close();
>
>
>
> Thanks
>
>
> Olivier
>