Purging unused segments.

[Erick Erickson <er...@gmail.com>]

I have a situation in which I can't safely optimize, the index is on a
machine that doesn't have enough disk space. I have no control over the
hardware. There are some raw statistics on this index that don't make
sense, it's roughly twice the size of a similar index (could be legit, but
it seems out of whack). And the number of deleted documents isn't big
enough to account for the discrepancy.

So is there a good way, without optimizing, to purge any segments not
referenced in the segments file? Actually I doubt that optimizing would
even do it if I _could_, any phantom segments aren't visible from the
segments file anyway...

I can generate a list of active segments using CheckIndex and compare that
against the contents of the index directory right enough, and remove the
segments that aren't listed, feels feels awkward and error-prone though (of
course the solr server should be killed while doing _any_ of this).

Hmmm, I suppose I could replicate the index to another fresh Solr
instance.....

Thanks,
Erick

Re: Purging unused segments.

[Erick Erickson <er...@gmail.com>]

Robert:

Thanks a million, that'll teach me to grep for the obvious <G>...

It's not even clear (I'm working twice-removed) that there _are_
unused files. I'm grasping at straws here....

Thanks again,
Erick


On Fri, Aug 9, 2013 at 9:32 PM, Robert Muir <rc...@gmail.com> wrote:

> On Fri, Aug 9, 2013 at 7:48 PM, Erick Erickson <er...@gmail.com>
> wrote:
> >
> > So is there a good way, without optimizing, to purge any segments not
> > referenced in the segments file? Actually I doubt that optimizing would
> > even do it if I _could_, any phantom segments aren't visible from the
> > segments file anyway...
> >
>
> I dont know why you have these files (windows? deletion policy?) but
> maybe you are interested in this:
>
>
> http://lucene.apache.org/core/4_4_0/core/org/apache/lucene/index/IndexWriter.html#deleteUnusedFiles%28%29
>

Re: Purging unused segments.

[Robert Muir <rc...@gmail.com>]

On Fri, Aug 9, 2013 at 7:48 PM, Erick Erickson <er...@gmail.com> wrote:
>
> So is there a good way, without optimizing, to purge any segments not
> referenced in the segments file? Actually I doubt that optimizing would
> even do it if I _could_, any phantom segments aren't visible from the
> segments file anyway...
>

I dont know why you have these files (windows? deletion policy?) but
maybe you are interested in this:

http://lucene.apache.org/core/4_4_0/core/org/apache/lucene/index/IndexWriter.html#deleteUnusedFiles%28%29