You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by kiwi clive <ki...@yahoo.com> on 2012/10/26 19:50:04 UTC

Lucene 3.6.0 Index Size

Hello.

We have an index that when creted using lucene2.3.2, has a size of about 4G.

Creating the same index (with the same parameters) with lucene 3.6.0 results in an 11G index.

Could someone shed some light into why the index is so much larger, given the same data and the same parameters?

I realize this is a large version jump but a doubling in index size does not seem a step in the right direction to me ;-)

I am using cfs format.

Thanks,
Clive

Re: Lucene 3.6.0 Index Size

Posted by kiwi clive <ki...@yahoo.com>.
Hi Vitaly,

Your hunch is correct, yes there are unmerged segments leftover. However to get indexing throughput, we use multiple threads on the writer flushing to disk periodically, but the writer can stay open for some time (until the last thread terminates). However, after an optimize, the index is closed. Thanks for the advice, I need to revisit the merging section of the application.

Clive




________________________________
 From: Vitaly Funstein <vf...@gmail.com>
To: java-user@lucene.apache.org 
Sent: Friday, October 26, 2012 8:13 PM
Subject: Re: Lucene 3.6.0 Index Size
 
One thing to keep in mind is that the default merge policy has changed in
3.6 from 2.3.2 (I'm almost certain of that). So it's just a hunch but you
may have some unmerged segments left over at the end. Try calling
IndexWriter.close(true) after you're done indexing.

On Fri, Oct 26, 2012 at 10:50 AM, kiwi clive <ki...@yahoo.com> wrote:

> Hello.
>
> We have an index that when creted using lucene2.3.2, has a size of about
> 4G.
>
> Creating the same index (with the same parameters) with lucene 3.6.0
> results in an 11G index.
>
> Could someone shed some light into why the index is so much larger, given
> the same data and the same parameters?
>
> I realize this is a large version jump but a doubling in index size does
> not seem a step in the right direction to me ;-)
>
> I am using cfs format.
>
> Thanks,
> Clive
>

Re: Lucene 3.6.0 Index Size

Posted by Vitaly Funstein <vf...@gmail.com>.
One thing to keep in mind is that the default merge policy has changed in
3.6 from 2.3.2 (I'm almost certain of that). So it's just a hunch but you
may have some unmerged segments left over at the end. Try calling
IndexWriter.close(true) after you're done indexing.

On Fri, Oct 26, 2012 at 10:50 AM, kiwi clive <ki...@yahoo.com> wrote:

> Hello.
>
> We have an index that when creted using lucene2.3.2, has a size of about
> 4G.
>
> Creating the same index (with the same parameters) with lucene 3.6.0
> results in an 11G index.
>
> Could someone shed some light into why the index is so much larger, given
> the same data and the same parameters?
>
> I realize this is a large version jump but a doubling in index size does
> not seem a step in the right direction to me ;-)
>
> I am using cfs format.
>
> Thanks,
> Clive
>