You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Ryan McKinley <ry...@gmail.com> on 2007/11/10 22:01:55 UTC
restoring a corrupt index?
Using solr, we have been running an indexing process for a while and
when I checked on it today, it spits out an error:
java.lang.RuntimeException: java.io.FileNotFoundException:
/path/to/index/_cf9.fnm (No such file or directory)
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:584)
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:475)
Looking through the archives, it looks like we are up a creek.
Any thoughts on what could have caused this? The log files contains
some 'too many open files' errors, I can't tell if that corresponds with
when the index went bad though.
the startup script includes:
ulimit -n 100000
which seems generous, no?
it is a 22GB index, ls -l | wc shows 180K files (oh my)
So my questions:
1. Anything I can do to use this index while I rebuild another? (takes a
long time!)
2. Does the ulimit number explain how the index got corrupted? If so,
it seems like a problem.
thanks
ryan
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: restoring a corrupt index?
Posted by Michael McCandless <lu...@mikemccandless.com>.
> Would this help: https://issues.apache.org/jira/browse/LUCENE-1020
That should help here, but please proceed with caution: this tool is
very new, has only been tested on trunk indices, and is brutal in how
it recovers the index (it removes the entire segment if there is any
problem loading one of its files).
You can run it without "-fix" and it will report details & any errors
it in loading your index.
Mike
"Grant Ingersoll" <gs...@apache.org> wrote:
> Would this help: https://issues.apache.org/jira/browse/LUCENE-1020
>
>
> On Nov 10, 2007, at 4:01 PM, Ryan McKinley wrote:
>
> > Using solr, we have been running an indexing process for a while and
> > when I checked on it today, it spits out an error:
> >
> > java.lang.RuntimeException: java.io.FileNotFoundException: /path/to/
> > index/_cf9.fnm (No such file or directory)
> > at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:584)
> > at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:475)
> >
> > Looking through the archives, it looks like we are up a creek.
> >
> > Any thoughts on what could have caused this? The log files contains
> > some 'too many open files' errors, I can't tell if that corresponds
> > with when the index went bad though.
> >
> > the startup script includes:
> > ulimit -n 100000
> > which seems generous, no?
> >
> > it is a 22GB index, ls -l | wc shows 180K files (oh my)
> >
> > So my questions:
> >
> > 1. Anything I can do to use this index while I rebuild another?
> > (takes a long time!)
> >
> > 2. Does the ulimit number explain how the index got corrupted? If
> > so, it seems like a problem.
> >
> >
> > thanks
> > ryan
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
>
> --------------------------
> Grant Ingersoll
> http://lucene.grantingersoll.com
>
> Lucene Boot Camp Training:
> ApacheCon Atlanta, Nov. 12, 2007. Sign up now! http://www.apachecon.com
>
> Lucene Helpful Hints:
> http://wiki.apache.org/lucene-java/BasicsOfPerformance
> http://wiki.apache.org/lucene-java/LuceneFAQ
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: restoring a corrupt index?
Posted by Grant Ingersoll <gs...@apache.org>.
Would this help: https://issues.apache.org/jira/browse/LUCENE-1020
On Nov 10, 2007, at 4:01 PM, Ryan McKinley wrote:
> Using solr, we have been running an indexing process for a while and
> when I checked on it today, it spits out an error:
>
> java.lang.RuntimeException: java.io.FileNotFoundException: /path/to/
> index/_cf9.fnm (No such file or directory)
> at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:584)
> at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:475)
>
> Looking through the archives, it looks like we are up a creek.
>
> Any thoughts on what could have caused this? The log files contains
> some 'too many open files' errors, I can't tell if that corresponds
> with when the index went bad though.
>
> the startup script includes:
> ulimit -n 100000
> which seems generous, no?
>
> it is a 22GB index, ls -l | wc shows 180K files (oh my)
>
> So my questions:
>
> 1. Anything I can do to use this index while I rebuild another?
> (takes a long time!)
>
> 2. Does the ulimit number explain how the index got corrupted? If
> so, it seems like a problem.
>
>
> thanks
> ryan
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
--------------------------
Grant Ingersoll
http://lucene.grantingersoll.com
Lucene Boot Camp Training:
ApacheCon Atlanta, Nov. 12, 2007. Sign up now! http://www.apachecon.com
Lucene Helpful Hints:
http://wiki.apache.org/lucene-java/BasicsOfPerformance
http://wiki.apache.org/lucene-java/LuceneFAQ
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: restoring a corrupt index?
Posted by Michael McCandless <lu...@mikemccandless.com>.
"Yonik Seeley" <yo...@apache.org> wrote:
> On Nov 11, 2007 1:59 PM, Michael McCandless <lu...@mikemccandless.com>
> wrote:
> > Ryan can you share any details of how you (Solr) is using Lucene? Are
> > you using autoCommit=false? I'd really love to get to the root cause
> > here.
>
> Unfortunately, Solr and Lucene both have something called autocommit now.
> Solr autocommit automatically called a solr level "commit" after a
> certain amount of time or docs were added (commit == close the writer,
> deleted buffered deletes, open new searcher to make the changes
> visible.) Lucene's autocommit (added in 2.3 AFAIK) if true, means that
> a new segment descriptor will be written (making it visible to a new
> reader opened on the index) each time a new segment is flushed by the
> writer... otherwise close() needs to be called for this to happen.
>
> Anyway, Ryan was using Lucene 2.2 when this happened (hence
> lucene_autoCommit==true, right?)
Actually Lucene autoCommit was first available in 2.2, but it sounds
like Solr leaves that at the default (true).
Mike
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: restoring a corrupt index?
Posted by Yonik Seeley <yo...@apache.org>.
On Nov 11, 2007 1:59 PM, Michael McCandless <lu...@mikemccandless.com> wrote:
> Ryan can you share any details of how you (Solr) is using Lucene? Are
> you using autoCommit=false? I'd really love to get to the root cause
> here.
Unfortunately, Solr and Lucene both have something called autocommit now.
Solr autocommit automatically called a solr level "commit" after a
certain amount of time or docs were added (commit == close the writer,
deleted buffered deletes, open new searcher to make the changes
visible.) Lucene's autocommit (added in 2.3 AFAIK) if true, means that
a new segment descriptor will be written (making it visible to a new
reader opened on the index) each time a new segment is flushed by the
writer... otherwise close() needs to be called for this to happen.
Anyway, Ryan was using Lucene 2.2 when this happened (hence
lucene_autoCommit==true, right?)
-Yonik
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: restoring a corrupt index?
Posted by Michael McCandless <lu...@mikemccandless.com>.
"vivek sar" <vi...@gmail.com> wrote:
> I think if the indexer is abruptly stopped while it's in progress
> the index corruption can happen.
One correction here: as far as I know, the index should not become
corrupt if the JVM is kill -9'd or JVM crashes. If that seems to be
happening then we need to dig down to the root cause and fix it...
However, if the OS or machine crashes, or power is pulled from the
machine, it is possible for the index to become corrupt. But for the
next version of Lucene (2.3) we are making changes to try to
ensure consistency even for these cases as well. See here for details:
https://issues.apache.org/jira/browse/LUCENE-1044
Mike
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: restoring a corrupt index?
Posted by vivek sar <vi...@gmail.com>.
We have seen similar exceptions (with Lucene 2.2) when were doing the
following mistakes,
1) Not closing the old searchers and re-creating a new one for every
new search (fixed it by closing the searcher every time, if you want
you could only one searcher instance as well)
2) Not having any jvm shutdown hook (in cases the JVM gets killled we
need to make sure the indexer is closed if it's in the middle of
indexing) - of course you would still need to avoid forceful killing
(kill -9).
Since, these two fixes we haven't seen this problem happening. I think
if the indexer is abruptly stopped while it's in progress the index
corruption can happen.
-vivek
On Nov 11, 2007 2:10 PM, Ryan McKinley <ry...@gmail.com> wrote:
>
> >
> > Ryan can you post the output of CheckIndex on your now-working index?
> > (1800 is still too many files I think, certainly after having
> > optimized).
> >
>
> ok, 1800 was wrong - that was from a botched attempt where I:
> 1. ran optimize on the broken 18K file index. It crashed midway through.
> 2. run CheckIndex -fix on that
>
> When I:
> 1. run CheckIndex -fix
> 2. run Optimize
> it results in an index with 65 files (that seems normal)
>
>
> > Also, what steps finally allowed you to recover? CheckIndex
> > (back-ported to 2.2) followed by optimize?
> >
>
> I did not back-port to 2.2, this converted 2.2 index to 2.3, and i
> shoved 2.3 libs into an exploded solr.war. (just to see if it works)
> It works, but I'm better off going back to an older working version.
>
>
> >
> > I'm still baffled how Lucene 2.2 could ever produce a corrupt index
> > even on hitting descriptor limits or other exceptions. I can see that
> > this could cause files to not be deleted properly, but, I can't see
> > how it can corrupt the index.
> >
>
> I'm not confident it is lucene's fault - the hardware has been flaky
> too. But the 18K files and 'too many open files' error makes me
> suspicious. Unfortunately, I can't quite grock when stuff started going
> wrong and how long it has been going on from my log files.
>
>
> > Ryan can you share any details of how you (Solr) is using Lucene? Are
> > you using autoCommit=false? I'd really love to get to the root cause
> > here.
> >
>
> I am using all standard solr config (copied from the example). I am
> using solr's <autoCommit> to "commit" added documents every 30 secs.
>
>
> ryan
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: restoring a corrupt index?
Posted by Ryan McKinley <ry...@gmail.com>.
>
> Ryan can you post the output of CheckIndex on your now-working index?
> (1800 is still too many files I think, certainly after having
> optimized).
>
ok, 1800 was wrong - that was from a botched attempt where I:
1. ran optimize on the broken 18K file index. It crashed midway through.
2. run CheckIndex -fix on that
When I:
1. run CheckIndex -fix
2. run Optimize
it results in an index with 65 files (that seems normal)
> Also, what steps finally allowed you to recover? CheckIndex
> (back-ported to 2.2) followed by optimize?
>
I did not back-port to 2.2, this converted 2.2 index to 2.3, and i
shoved 2.3 libs into an exploded solr.war. (just to see if it works)
It works, but I'm better off going back to an older working version.
>
> I'm still baffled how Lucene 2.2 could ever produce a corrupt index
> even on hitting descriptor limits or other exceptions. I can see that
> this could cause files to not be deleted properly, but, I can't see
> how it can corrupt the index.
>
I'm not confident it is lucene's fault - the hardware has been flaky
too. But the 18K files and 'too many open files' error makes me
suspicious. Unfortunately, I can't quite grock when stuff started going
wrong and how long it has been going on from my log files.
> Ryan can you share any details of how you (Solr) is using Lucene? Are
> you using autoCommit=false? I'd really love to get to the root cause
> here.
>
I am using all standard solr config (copied from the example). I am
using solr's <autoCommit> to "commit" added documents every 30 secs.
ryan
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: restoring a corrupt index?
Posted by Michael McCandless <lu...@mikemccandless.com>.
"Yonik Seeley" <yo...@apache.org> wrote:
> On Nov 11, 2007 12:48 PM, Ryan McKinley <ry...@gmail.com> wrote:
> > > Ryan are you able to update to that commit I just did? If so I think
> > > you should run the tool without -fix and post back what it printed. It
> > > should report an error on that one segment due to the missing file.
> > > Then, run -fix to remove that segment (please backup your index first!).
> > > Then, if you have a zillion segments in the index, try optimizing it?
> > >
> >
> > uggg. Thanks for all your help / pointers. I've been able to salvage a
> > functioning index - (with 800K fewer docs) - I'll just go back to an old
> > index and build from there.
> >
> > Optimizing reduced 180K files to 1800 - I'm guessing that had something
> > to do with why it hit a max open file limit.
> It shouldn't. A single lucene segment should have about 8 or 9 files
> associated with it, so I don't know what the remaining 1800 are. An
> index (with normal settings) should *never* get to 180K files that are
> actually part of the index.
Ryan can you post the output of CheckIndex on your now-working index?
(1800 is still too many files I think, certainly after having
optimized).
Also, what steps finally allowed you to recover? CheckIndex
(back-ported to 2.2) followed by optimize?
> > If I understand Yonik's comments, something in lucene 2.3 has changed so
> > that hitting the max open file limit could not result in a corrupt
> > index. Is that true?
>
> I think Lucene has become more resistant to index corruption on a
> failure (like an exception).
> Just make sure you do a commit once in a while (old versions of lucene
> flushed a new segment and it was automatically added into the current
> view of the index... the newest version of lucene with
> lucene_autocommit=off won't do that).
I'm still baffled how Lucene 2.2 could ever produce a corrupt index
even on hitting descriptor limits or other exceptions. I can see that
this could cause files to not be deleted properly, but, I can't see
how it can corrupt the index.
Ryan can you share any details of how you (Solr) is using Lucene? Are
you using autoCommit=false? I'd really love to get to the root cause
here.
Mike
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: restoring a corrupt index?
Posted by Yonik Seeley <yo...@apache.org>.
On Nov 11, 2007 12:48 PM, Ryan McKinley <ry...@gmail.com> wrote:
> > Ryan are you able to update to that commit I just did? If so I think
> > you should run the tool without -fix and post back what it printed. It
> > should report an error on that one segment due to the missing file.
> > Then, run -fix to remove that segment (please backup your index first!).
> > Then, if you have a zillion segments in the index, try optimizing it?
> >
>
> uggg. Thanks for all your help / pointers. I've been able to salvage a
> functioning index - (with 800K fewer docs) - I'll just go back to an old
> index and build from there.
>
> Optimizing reduced 180K files to 1800 - I'm guessing that had something
> to do with why it hit a max open file limit.
It shouldn't. A single lucene segment should have about 8 or 9 files
associated with it, so I don't know what the remaining 1800 are. An
index (with normal settings) should *never* get to 180K files that are
actually part of the index.
> I'm running a standard solr-trunk from a few weeks back (nothing custom,
> no 'embedded' stuff). The index is constantly updating a few files, so
> I am not surprised by the mass of unused files optimize cleans up.
All files that look like segment files are cleaned up when they are
not referenced now.
The fact that optimize cleaned them up doesn't mean they were in use before.
> If I understand Yonik's comments, something in lucene 2.3 has changed so
> that hitting the max open file limit could not result in a corrupt
> index. Is that true?
I think Lucene has become more resistant to index corruption on a
failure (like an exception).
Just make sure you do a commit once in a while (old versions of lucene
flushed a new segment and it was automatically added into the current
view of the index... the newest version of lucene with
lucene_autocommit=off won't do that).
> In the meantime would optimizing more often help
> avoid this?
I think the root cause is more running out of descriptors. It may be
that you just need a higher limit.
If you are using Linux, see this link for how to check/set the number
of system-wide descriptors:
http://www.cs.uwaterloo.ca/~brecht/servers/openfiles.html
-Yonik
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: restoring a corrupt index?
Posted by Ryan McKinley <ry...@gmail.com>.
>
> Ryan are you able to update to that commit I just did? If so I think
> you should run the tool without -fix and post back what it printed. It
> should report an error on that one segment due to the missing file.
> Then, run -fix to remove that segment (please backup your index first!).
> Then, if you have a zillion segments in the index, try optimizing it?
>
uggg. Thanks for all your help / pointers. I've been able to salvage a
functioning index - (with 800K fewer docs) - I'll just go back to an old
index and build from there.
Optimizing reduced 180K files to 1800 - I'm guessing that had something
to do with why it hit a max open file limit.
I'm running a standard solr-trunk from a few weeks back (nothing custom,
no 'embedded' stuff). The index is constantly updating a few files, so
I am not surprised by the mass of unused files optimize cleans up.
If I understand Yonik's comments, something in lucene 2.3 has changed so
that hitting the max open file limit could not result in a corrupt
index. Is that true? In the meantime would optimizing more often help
avoid this?
thanks
ryan
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: restoring a corrupt index?
Posted by Michael McCandless <lu...@mikemccandless.com>.
"Yonik Seeley" <yo...@apache.org> wrote:
> On Nov 10, 2007 5:01 PM, Michael McCandless <lu...@mikemccandless.com>
> wrote:
> > "Yonik Seeley" <yo...@apache.org> wrote:
> >
> > > > How can this lead to index corruption? The "no such file or directory" on
> > > > loading _cf9.fnm sounds like index corruption?
> > >
> > > I don't think older versions of lucene handled these errors as well.
> > > Perhaps _cf9.fnm failed to be written, but the segments file succeeded.
> > > It could also be Solr's fault for allowing further operations on an
> > > index after one failed? I'm not sure how that should be handled.
> >
> > Ahh, OK.
> >
> > Yeah it's not clear how Solr should handle this case, though Lucene
> > should be at (get to?) the point where on exception no harm to the
> > index can be done. Ie, index on disk is left consistent, and, the
> > state of the writer is such that it can't corrupt the index even if
> > it's further used after the exception.
>
> Right... keep in mind concurrent updates too... it's pretty much
> impossible for the user-level to avoid other calls to addDocument()
> from going ahead since they can be in different threads.
Ahh, true.
> > OK, though, writer doesn't use many descriptors at all, right?
>
> Yeah, but other things might be usng up the descriptors... there is
> probably an open searcher for searching, possibly another one warming
> in the background, plus whatever else is going on in the servlet
> container.
OK.
> Ryan, are you using stock solr, or embedded? Anything that might
> prevent old searchers from being closed (you chould check the log
> files for #opens vs #closes)
>
> > It
> > opens 1 segment's worth to flush, and mergeFactor+1 segment's worth to
> > merge. It's spooky if Lucene can create zillions of un-referenced
> > files.
>
> Perhaps unreferenced files may not be deleted if an exception is
> encountered first, or perhaps even the deleter is failing due to lack
> of descriptors.
OK. Still spooky that this failure mode is possible on 2.2.
> > Would be good to get to the root cause & make sure it's really
> > fixed on trunk.
>
> Is there a tool we could have ryan point at the segments file and get
> a dump of the referenced segments?
LUCENE-1020 will print the details of all segments ... hmmm, I see
that I'm not closing each SegmentReader in that tool. OK I just committed
a fix. Otherwise I think Ryan would hit file descriptor limit (if these
files are referenced).
Ryan are you able to update to that commit I just did? If so I think
you should run the tool without -fix and post back what it printed. It
should report an error on that one segment due to the missing file.
Then, run -fix to remove that segment (please backup your index first!).
Then, if you have a zillion segments in the index, try optimizing it?
Oh, hang on: you can't run -fix because this tool will write a
segments_N file in the current trunk format, which is different
from 2.2.x. It would not be hard to port the tool back to 2.2:
just comment out those System.out.println's that hit compilation
errors.
> > Or, maybe they are all referenced -- I think there were issues with
> > the old merge policy that could cause segments to not be merged when
> > they should have been?
>
> That should not have been the case in Lucene 2.2 I think.
Ahh, right.
Mike
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: restoring a corrupt index?
Posted by Yonik Seeley <yo...@apache.org>.
On Nov 10, 2007 5:01 PM, Michael McCandless <lu...@mikemccandless.com> wrote:
> "Yonik Seeley" <yo...@apache.org> wrote:
>
> > > How can this lead to index corruption? The "no such file or directory" on
> > > loading _cf9.fnm sounds like index corruption?
> >
> > I don't think older versions of lucene handled these errors as well.
> > Perhaps _cf9.fnm failed to be written, but the segments file succeeded.
> > It could also be Solr's fault for allowing further operations on an
> > index after one failed? I'm not sure how that should be handled.
>
> Ahh, OK.
>
> Yeah it's not clear how Solr should handle this case, though Lucene
> should be at (get to?) the point where on exception no harm to the
> index can be done. Ie, index on disk is left consistent, and, the
> state of the writer is such that it can't corrupt the index even if
> it's further used after the exception.
Right... keep in mind concurrent updates too... it's pretty much
impossible for the user-level to avoid other calls to addDocument()
from going ahead since they can be in different threads.
> OK, though, writer doesn't use many descriptors at all, right?
Yeah, but other things might be usng up the descriptors... there is
probably an open searcher for searching, possibly another one warming
in the background, plus whatever else is going on in the servlet
container.
Ryan, are you using stock solr, or embedded? Anything that might
prevent old searchers from being closed (you chould check the log
files for #opens vs #closes)
> It
> opens 1 segment's worth to flush, and mergeFactor+1 segment's worth to
> merge. It's spooky if Lucene can create zillions of un-referenced
> files.
Perhaps unreferenced files may not be deleted if an exception is
encountered first, or perhaps even the deleter is failing due to lack
of descriptors.
> Would be good to get to the root cause & make sure it's really
> fixed on trunk.
Is there a tool we could have ryan point at the segments file and get
a dump of the referenced segments?
> Or, maybe they are all referenced -- I think there were issues with
> the old merge policy that could cause segments to not be merged when
> they should have been?
That should not have been the case in Lucene 2.2 I think.
-Yonik
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: restoring a corrupt index?
Posted by Ryan McKinley <ry...@gmail.com>.
>
>>> Or maybe the index is not corrupt but then we are hitting the descriptor limit
>>> on opening a searcher and it's being reported as "no such file or directory"?
>> Hmmm, yes that's possible too...
>> Should be easy to tell by checking if the file _cf9.fnm already exists.
>
> Oh yeah. Ryan, does that file exist?
>
no luck.
ryan@posh:~$ ls -l /array/home/nestor/data2/flat/index/_cf9.fnm
ls: /array/home/nestor/data2/flat/index/_cf9.fnm: No such file or directory
I'm backing up the broken index now and will try:
1. run optimize and hope for the best
2. LUCENE-1020
and then hope for the best.
The index takes ~week to build so even having something with a block of
stuff missing is better then zero (or gulp, our last backup)
thanks
ryan
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: restoring a corrupt index?
Posted by Michael McCandless <lu...@mikemccandless.com>.
"Yonik Seeley" <yo...@apache.org> wrote:
> > How can this lead to index corruption? The "no such file or directory" on
> > loading _cf9.fnm sounds like index corruption?
>
> I don't think older versions of lucene handled these errors as well.
> Perhaps _cf9.fnm failed to be written, but the segments file succeeded.
> It could also be Solr's fault for allowing further operations on an
> index after one failed? I'm not sure how that should be handled.
Ahh, OK.
Yeah it's not clear how Solr should handle this case, though Lucene
should be at (get to?) the point where on exception no harm to the
index can be done. Ie, index on disk is left consistent, and, the
state of the writer is such that it can't corrupt the index even if
it's further used after the exception.
> > Or maybe the index is not corrupt but then we are hitting the descriptor limit
> > on opening a searcher and it's being reported as "no such file or directory"?
>
> Hmmm, yes that's possible too...
> Should be easy to tell by checking if the file _cf9.fnm already exists.
Oh yeah. Ryan, does that file exist?
> > Yonik do you understand why so many unreferenced files are being produced
> > here? What's the root cause?
>
> Just guesses... but perhaps new index files are written but things
> fail before the stage where old files are deleted?
OK, though, writer doesn't use many descriptors at all, right? It
opens 1 segment's worth to flush, and mergeFactor+1 segment's worth to
merge. It's spooky if Lucene can create zillions of un-referenced
files. Would be good to get to the root cause & make sure it's really
fixed on trunk.
Or, maybe they are all referenced -- I think there were issues with
the old merge policy that could cause segments to not be merged when
they should have been?
Mike
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: restoring a corrupt index?
Posted by Ryan McKinley <ry...@gmail.com>.
thanks for all the replies
>> Yonik do you understand why so many unreferenced files are being produced
>> here? What's the root cause?
>
This is an index that has the same documents get updated many times,
that could build up old files w/o optimizing.
> Just guesses... but perhaps new index files are written but things
> fail before the stage where old files are deleted?
>
> Ryan, what version of lucene is this version of Solr using?
>
solr trunk from a few weeks back. packaged lucene jar is lucene-core-2.2.0
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: restoring a corrupt index?
Posted by Yonik Seeley <yo...@apache.org>.
On Nov 10, 2007 4:34 PM, Michael McCandless <lu...@mikemccandless.com> wrote:
>
> "Yonik Seeley" <yo...@apache.org> wrote:
> > On Nov 10, 2007 4:01 PM, Ryan McKinley <ry...@gmail.com> wrote:
> > > Using solr, we have been running an indexing process for a while and
> > > when I checked on it today, it spits out an error:
> > >
> > > java.lang.RuntimeException: java.io.FileNotFoundException:
> > > /path/to/index/_cf9.fnm (No such file or directory)
> > > at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:584)
> > > at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:475)
> > >
> > > Looking through the archives, it looks like we are up a creek.
> > >
> > > Any thoughts on what could have caused this? The log files contains
> > > some 'too many open files' errors, I can't tell if that corresponds with
> > > when the index went bad though.
> >
> > Yup... that would most likely be it.
>
> How can this lead to index corruption? The "no such file or directory" on
> loading _cf9.fnm sounds like index corruption?
I don't think older versions of lucene handled these errors as well.
Perhaps _cf9.fnm failed to be written, but the segments file succeeded.
It could also be Solr's fault for allowing further operations on an
index after one failed? I'm not sure how that should be handled.
> Or maybe the index is not corrupt but then we are hitting the descriptor limit
> on opening a searcher and it's being reported as "no such file or directory"?
Hmmm, yes that's possible too...
Should be easy to tell by checking if the file _cf9.fnm already exists.
> If so, maybe opening an IndexWriter and optimizing the index would recover
> it? (Make a backup copy of the index first!).
>
> > > 2. Does the ulimit number explain how the index got corrupted? If so,
> > > it seems like a problem.
> >
> > I think newest lucene versions would prevent this with
> > lucene_autocommit=false. A new segments file (the file which
> > references all other files in the current index) is not written until
> > a close of the writer.
>
> Yonik do you understand why so many unreferenced files are being produced
> here? What's the root cause?
Just guesses... but perhaps new index files are written but things
fail before the stage where old files are deleted?
Ryan, what version of lucene is this version of Solr using?
-Yonik
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: restoring a corrupt index?
Posted by Michael McCandless <lu...@mikemccandless.com>.
"Yonik Seeley" <yo...@apache.org> wrote:
> On Nov 10, 2007 4:01 PM, Ryan McKinley <ry...@gmail.com> wrote:
> > Using solr, we have been running an indexing process for a while and
> > when I checked on it today, it spits out an error:
> >
> > java.lang.RuntimeException: java.io.FileNotFoundException:
> > /path/to/index/_cf9.fnm (No such file or directory)
> > at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:584)
> > at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:475)
> >
> > Looking through the archives, it looks like we are up a creek.
> >
> > Any thoughts on what could have caused this? The log files contains
> > some 'too many open files' errors, I can't tell if that corresponds with
> > when the index went bad though.
>
> Yup... that would most likely be it.
How can this lead to index corruption? The "no such file or directory" on
loading _cf9.fnm sounds like index corruption?
Or maybe the index is not corrupt but then we are hitting the descriptor limit
on opening a searcher and it's being reported as "no such file or directory"?
If so, maybe opening an IndexWriter and optimizing the index would recover
it? (Make a backup copy of the index first!).
> > 2. Does the ulimit number explain how the index got corrupted? If so,
> > it seems like a problem.
>
> I think newest lucene versions would prevent this with
> lucene_autocommit=false. A new segments file (the file which
> references all other files in the current index) is not written until
> a close of the writer.
Yonik do you understand why so many unreferenced files are being produced
here? What's the root cause?
Mike
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: restoring a corrupt index?
Posted by Yonik Seeley <yo...@apache.org>.
On Nov 10, 2007 4:01 PM, Ryan McKinley <ry...@gmail.com> wrote:
> Using solr, we have been running an indexing process for a while and
> when I checked on it today, it spits out an error:
>
> java.lang.RuntimeException: java.io.FileNotFoundException:
> /path/to/index/_cf9.fnm (No such file or directory)
> at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:584)
> at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:475)
>
> Looking through the archives, it looks like we are up a creek.
>
> Any thoughts on what could have caused this? The log files contains
> some 'too many open files' errors, I can't tell if that corresponds with
> when the index went bad though.
Yup... that would most likely be it.
> the startup script includes:
> ulimit -n 100000
> which seems generous, no?
The kernel may have a lower limit.
> it is a 22GB index, ls -l | wc shows 180K files (oh my)
I don't think any index with a normal mergeFactor should have that many files.
Most of these files are probably unreferenced by the current index but
haven't been cleaned up due to the errors with file descriptors.
> So my questions:
>
> 1. Anything I can do to use this index while I rebuild another? (takes a
> long time!)
Doubt it... you would never be sure if the index was correct.
> 2. Does the ulimit number explain how the index got corrupted? If so,
> it seems like a problem.
I think newest lucene versions would prevent this with
lucene_autocommit=false. A new segments file (the file which
references all other files in the current index) is not written until
a close of the writer.
-Yonik
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org