You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Chris Harris <ry...@gmail.com> on 2008/08/14 20:37:29 UTC
More files in index directory than expected
It's my understanding that if my mergeFactor is 10, then there
shouldn't be more than 11 segments in my index directory (10 segments,
plus an additional segment if a merge is in progress). It would seem
to follow that there shouldn't be more than 11 fdt files, 11 tis
files, etc.. However, I'm looking at one of my indexes now, and this
doesn't seem to be the case. Here are the tis files for this index,
for instance:
07/22/2008 07:49 PM 77,925,180 _1je.tis
07/23/2008 02:57 AM 65,988,651 _256.tis
07/23/2008 04:18 AM 13,159,578 _29t.tis
07/23/2008 05:08 AM 10,146,941 _2cw.tis
07/23/2008 05:39 AM 6,749,665 _2el.tis
07/23/2008 06:24 AM 12,274,012 _2he.tis
07/23/2008 07:01 AM 14,069,531 _2kh.tis
07/23/2008 07:53 AM 13,795,213 _2nu.tis
07/23/2008 08:20 AM 6,284,902 _2p0.tis
07/23/2008 08:27 AM 1,980,945 _2p9.tis
07/23/2008 08:36 AM 1,674,640 _2pk.tis
07/23/2008 08:37 AM 311,483 _2pl.tis
07/23/2008 08:38 AM 285,881 _2pm.tis
07/23/2008 08:39 AM 245,138 _2pn.tis
07/23/2008 08:40 AM 116,881 _2po.tis
07/17/2008 11:22 PM 69,635,905 _rp.tis
07/18/2008 12:59 AM 15,883,866 _xu.tis
There are 17 of these files. (File sizes are in bytes.) When I open up
the index in Luke, it says all of them are "In Use" and it doesn't
list any of them as "Deletable". This seems to rule out the
possibility that Solr/Lucene somehow "forget" to clean up files that
were no longer in use.
I'm noticing that _2pk, _2pl, _2pm, _2pn, _2po are sequential file
names, alphabetically speaking, and their last modified times are very
close to one another. Does this mean they're actually part of the same
segment, even though they are in separate files? If those files are
indeed part of a single segment, then the number of segments
represented by these files would really be 17-4=13. But that's still
more than the expected 11 segments.
I just discovered that one of my other indexes has over 11,000 tis
files. That's disturbing. I'm not sure if it would have the same
underlying cause.
Any ideas?
Re: More files in index directory than expected
Posted by Mark Miller <ma...@gmail.com>.
> The main thing that bugs me about this index now is that the latest
> version of Luke (0.8.1) won't open it. ("Unknown format version: -6")
> The Solr Luke handler works fine with it, though.
>
Luke comes with a released version of Lucene probably, while solr is
using a later version. You have to start luke with the solr Lucene jar
on the classpath. There are directions to do this type of thing on the
Luke webpage.
- Mark
Re: More files in index directory than expected
Posted by Yonik Seeley <yo...@apache.org>.
On Thu, Aug 14, 2008 at 6:31 PM, Chris Harris <ry...@gmail.com> wrote:
> (The only time a
> segment will be modified is if you delete files from it, and that will
> only alter the segment's .del file, leaving .tis and friends alone.)
Actually, these days .del files are even versioned.
> I don't know exactly what happened, but I restarted Solr once or twice
> and then when I started adding documents again, Solr started deleting
> segment files, and brought things down from like 500GB to like 18GB. I
> feel like I read somewhere that Solr sometimes has trouble deleting
> segment files when running on Windows. (I'm on Windows right now.) I
> wonder if that's related.
Right... file deletion just tends to be delayed a bit longer on Windows.
For example, if you do an optimize, all the segments will be merged
into a single segment, but because a reader is still holding open the
old index, you will see both sets of files. The old set of files will
be deleted when a new IndexWriter is opened after the old reader is
closed.
-Yonik
Re: More files in index directory than expected
Posted by Chris Harris <ry...@gmail.com>.
On Thu, Aug 14, 2008 at 2:01 PM, Michael McCandless
<lu...@mikemccandless.com> wrote:
> Chris Harris <ry...@gmail.com> wrote:
>> It's my understanding that if my mergeFactor is 10, then there
>> shouldn't be more than 11 segments in my index directory (10 segments,
>> plus an additional segment if a merge is in progress).
>
> Actually, mergeFactor 10 means each *level* will have <= 10 segments,
> where a level is roughly 10X the size of the previous level.
>
> EG after 10 segments (level 0) are flushed, they get merged into a
> single level 1 segment. Another 10 produces another level 1 segment.
> Etc. Until you have 10 level 1 segments, which then get merged into a
> single level 2 segment.
>
> The number of levels you have is logarithmic in the size of your index.
Thanks, that undoes a lot of my confusion.
As for segment creation, is it accurate to say the following: Solr
will write a new level 0 segment to disk each time an additional
ramBufferSizeMB (default=32MB) worth of data have been added to the
index. Furthermore, once that 32MB worth of data has been written to
disk, those segment's files will never be modified. (The only time a
segment will be modified is if you delete files from it, and that will
only alter the segment's .del file, leaving .tis and friends alone.)
>> I just discovered that one of my other indexes has over 11,000 tis
>> files. That's disturbing. I'm not sure if it would have the same
>> underlying cause.
>
> That does NOT sound right. Can you provide more details how this
> index is created/maintained?
I don't know exactly what happened, but I restarted Solr once or twice
and then when I started adding documents again, Solr started deleting
segment files, and brought things down from like 500GB to like 18GB. I
feel like I read somewhere that Solr sometimes has trouble deleting
segment files when running on Windows. (I'm on Windows right now.) I
wonder if that's related.
The main thing that bugs me about this index now is that the latest
version of Luke (0.8.1) won't open it. ("Unknown format version: -6")
The Solr Luke handler works fine with it, though.
Re: More files in index directory than expected
Posted by Michael McCandless <lu...@mikemccandless.com>.
Chris Harris <ry...@gmail.com> wrote:
> It's my understanding that if my mergeFactor is 10, then there
> shouldn't be more than 11 segments in my index directory (10 segments,
> plus an additional segment if a merge is in progress).
Actually, mergeFactor 10 means each *level* will have <= 10 segments,
where a level is roughly 10X the size of the previous level.
EG after 10 segments (level 0) are flushed, they get merged into a
single level 1 segment. Another 10 produces another level 1 segment.
Etc. Until you have 10 level 1 segments, which then get merged into a
single level 2 segment.
The number of levels you have is logarithmic in the size of your index.
> I'm noticing that _2pk, _2pl, _2pm, _2pn, _2po are sequential file
> names, alphabetically speaking, and their last modified times are very
> close to one another. Does this mean they're actually part of the same
> segment, even though they are in separate files?
No, these are different segments, just flushed shortly after one
another in time.
> I just discovered that one of my other indexes has over 11,000 tis
> files. That's disturbing. I'm not sure if it would have the same
> underlying cause.
That does NOT sound right. Can you provide more details how this
index is created/maintained?
Mike