You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Wang Guangchen <gu...@gmail.com> on 2009/05/25 10:42:04 UTC
Recover crashed solr index
Hi everyone,
I have 8m docs to index, and each doc is around 50kb. The solr crashed in
the middle of indexing. error message said that one of the file in the data
directory is missing. I don't know why this is happened.
So right now I have to find a way to recover the index to avoid re-index. Is
there anyone know any tools or method to recover the crashed index? Please
help.
Thanks a lot.
Regards
GC
Re: Recover crashed solr index
Posted by Wang Guangchen <gu...@gmail.com>.
hi Mike and Yonik ,
There is no other exceptions other than missing segments_cje. the disk still
have 160G more spaces when the disk crash.
current these is only one solr core that accessing this index directory. so
it can't be the file lock problem.
I am not sure is considered as unusual configuration, so i attached the
settings that i changed from the original solrconfig.xml.
<indexDefaults>
<ramBufferSizeMB>256</ramBufferSizeMB>
<maxMergeDocs>2147483647</maxMergeDocs>
<maxFieldLength>10000</maxFieldLength>
<writeLockTimeout>1000</writeLockTimeout>
<commitLockTimeout>10000</commitLockTimeout>
</indexDefaults>
<mainIndex>
<useCompoundFile>false</useCompoundFile>
<ramBufferSizeMB>128</ramBufferSizeMB>
<mergeFactor>15</mergeFactor><!-- default value : 10. Less value speed
up searching, but slow down indexing/ -->
<!-- Deprecated -->
<!--<maxBufferedDocs>1000</maxBufferedDocs>-->
<maxMergeDocs>2147483647</maxMergeDocs>
<maxFieldLength>10000</maxFieldLength>
<!-- If true, unlock any held write or commit locks on startup.
This defeats the locking mechanism that allows multiple
processes to safely access a lucene index, and should be
used with care.
This is not needed if lock type is 'none' or 'single'
-->
<unlockOnStartup>false</unlockOnStartup>
</mainIndex>
<updateHandler class="solr.DirectUpdateHandler2">
<autoCommit>
<maxDocs>1000</maxDocs>
</autoCommit>
</updateHandler>
Thanks a lot for your advice.
On Fri, May 29, 2009 at 5:44 AM, Yonik Seeley <yo...@lucidimagination.com>wrote:
> On Wed, May 27, 2009 at 5:49 AM, Michael McCandless
> <lu...@mikemccandless.com> wrote:
> > Hmm... so in fact it looks like Solr had done a number of commits
> > already (especially, given how large your generation is -- the "cje"
> > in segments_cje means there were a number of commits).
> >
> > Were there any other exceptions leading up to this? Disk full?
> > Anything unusual in your Solr configuration?
> >
> > Is there any chance that a 2nd Solr core attempted to access this same
> > directory?
>
> Solr uses SingleInstanceLockFactory by default... so Solr cores (in a
> multi-core env) would be protected.
> But maybe we should switch to NativeFSLockFactory to be safe from
> other processes (with the downside that it will fail for some
> filesystems).
>
> <!--
> This option specifies which Lucene LockFactory implementation to use.
>
> single = SingleInstanceLockFactory - suggested for a read-only index
> or when there is no possibility of another process trying
> to modify the index.
> native = NativeFSLockFactory
> simple = SimpleFSLockFactory
>
> (For backwards compatibility with Solr 1.2, 'simple' is the default
> if not specified.)
> -->
> <lockType>single</lockType>
>
>
> -Yonik
> http://www.lucidimagination.com
>
Re: Recover crashed solr index
Posted by Yonik Seeley <yo...@lucidimagination.com>.
On Wed, May 27, 2009 at 5:49 AM, Michael McCandless
<lu...@mikemccandless.com> wrote:
> Hmm... so in fact it looks like Solr had done a number of commits
> already (especially, given how large your generation is -- the "cje"
> in segments_cje means there were a number of commits).
>
> Were there any other exceptions leading up to this? Disk full?
> Anything unusual in your Solr configuration?
>
> Is there any chance that a 2nd Solr core attempted to access this same
> directory?
Solr uses SingleInstanceLockFactory by default... so Solr cores (in a
multi-core env) would be protected.
But maybe we should switch to NativeFSLockFactory to be safe from
other processes (with the downside that it will fail for some
filesystems).
<!--
This option specifies which Lucene LockFactory implementation to use.
single = SingleInstanceLockFactory - suggested for a read-only index
or when there is no possibility of another process trying
to modify the index.
native = NativeFSLockFactory
simple = SimpleFSLockFactory
(For backwards compatibility with Solr 1.2, 'simple' is the default
if not specified.)
-->
<lockType>single</lockType>
-Yonik
http://www.lucidimagination.com
Re: Recover crashed solr index
Posted by Michael McCandless <lu...@mikemccandless.com>.
Hmm... so in fact it looks like Solr had done a number of commits
already (especially, given how large your generation is -- the "cje"
in segments_cje means there were a number of commits).
Were there any other exceptions leading up to this? Disk full?
Anything unusual in your Solr configuration?
Is there any chance that a 2nd Solr core attempted to access this same
directory?
Mike
On Tue, May 26, 2009 at 9:33 PM, Wang Guangchen <gu...@gmail.com> wrote:
> Hi Mike,
>
> The index is autoCommit every 1000 docs. I set this to increase the indexing
> speed. What is the best configuration do you suggest for each commit cycle?
>
> Thank you very much for your help.
>
> Following is the original exception:
>
> java.lang.RuntimeException: java.io.FileNotFoundException:
> /solr/example/data/index/segments_cje (No such file or directory)
> at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1085)
> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:561)
> at
> org.apache.solr.core.CoreContainer$Initializer.initialize(CoreContainer.java:121)
>
> at
> org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:69)
> at
> org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:275)
> at
> org.apache.catalina.core.ApplicationFilterConfig.setFilterDef(ApplicationFilterConfig.java:397)
>
> at
> org.apache.catalina.core.ApplicationFilterConfig.<init>(ApplicationFilterConfig.java:108)
>
> at
> org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:3709)
> at
> org.apache.catalina.core.StandardContext.start(StandardContext.java:4363)
> at
> org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:791)
>
> at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:771)
> at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:525)
> at
> org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:627)
>
> at
> org.apache.catalina.startup.HostConfig.deployDescriptors(HostConfig.java:553)
> at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:488)
> at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1149)
> at
> org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:311)
> at
> org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:117)
>
> at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1053)
> at org.apache.catalina.core.StandardHost.start(StandardHost.java:719)
> at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1045)
> at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:443)
> at org.apache.catalina.core.StandardService.start(StandardService.java:516)
> at org.apache.catalina.core.StandardServer.start(StandardServer.java:710)
> at org.apache.catalina.startup.Catalina.start(Catalina.java:578)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288)
> at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413) Caused by:
> java.io.FileNotFoundException:
> /mnt_APS/solr/solrHomeFull/data/index/segments_cje (No such file or
> directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
> at
> org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor.<init>(FSDirectory.java:630)
> at
> org.apache.lucene.store.FSDirectory$FSIndexInput.<init>(FSDirectory.java:660)
>
> at
> org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.<init>(NIOFSDirectory.java:76)
>
> at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:63)
> at org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:560)
> at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:224)
> at
> org.apache.lucene.index.DirectoryIndexReader$1.doBody(DirectoryIndexReader.java:103)
>
> at
> org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:688)
>
> at
> org.apache.lucene.index.DirectoryIndexReader.open(DirectoryIndexReader.java:123)
> at org.apache.lucene.index.IndexReader.open(IndexReader.java:316)
> at org.apache.lucene.index.IndexReader.open(IndexReader.java:237)
> at org.apache.solr.search.SolrIndexSearcher.<init>
>
>
>
> On Wed, May 27, 2009 at 2:32 AM, Michael McCandless <
> lucene@mikemccandless.com> wrote:
>
>> It sounds like you had never committed while building the original
>> index? Unfortunately, it's not easy to recover an index in that
>> state. It's best to periodically commit if you're building such a
>> large index.
>>
>> Do you have the original exception you hit?
>>
>> I'll fix CheckIndex to be more sane if it could not load any segments file.
>>
>> Mike
>>
>> On Tue, May 26, 2009 at 2:12 AM, Wang Guangchen <gu...@gmail.com>
>> wrote:
>> > Hi peter,
>> >
>> > Thank you very much for your quick reply.
>> >
>> > I tried the CheckIndex method. It can't work on my crashed index.
>> > In the error message, it says the segments file in the directory is
>> missing.
>> > and when I use the -fix param, new segments file still can't be write.
>> > I even try the CheckIndex without the assertion, it still can't work.
>> >
>> >
>> > Do you know why this is happening ? Does it mean that the segment file
>> can't
>> > be rewrite at all?
>> >
>> > Btw, i am using the nightly build solr.
>> >
>> > following is the error messages:
>> >
>> > [root@localhost lib]# java -cp lucene-core-2.9-dev.jar
>> > org.apache.lucene.index.CheckIndex -fix /solr/example/data/index/
>> >
>> > NOTE: testing will be more thorough if you run java with
>> > '-ea:org.apache.lucene...', so assertions are enabled
>> >
>> > Opening index @ /solr/example/data/index/
>> >
>> > ERROR: could not read any segments file in directory
>> > java.io.FileNotFoundException: /solr/example
>> >
>> > /data/index/segments_cje (No such file or directory)
>> > at java.io.RandomAccessFile.open(Native Method)
>> > at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
>> > at
>> >
>> org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor.<init>(FSDirectory.java:630)
>> > at
>> >
>> org.apache.lucene.store.FSDirectory$FSIndexInput.<init>(FSDirectory.java:660)
>> > at
>> > org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:566)
>> > at
>> > org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:560)
>> > at
>> org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:224)
>> > at
>> > org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:292)
>> > at
>> >
>> org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:688)
>> > at
>> org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:289)
>> > at
>> > org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:258)
>> > at org.apache.lucene.index.CheckIndex.main(CheckIndex.java:678)
>> > WARNING: 0 documents will be lost
>> >
>> > NOTE: will write new segments file in 5 seconds; this will remove 0 docs
>> > from the index. THIS IS YOUR LAST CHANCE TO CTRL+C!
>> > 5...
>> > 4...
>> > 3...
>> > 2...
>> > 1...
>> > Writing...
>> > Exception in thread "main" java.lang.NullPointerException
>> > at
>> org.apache.lucene.index.CheckIndex.fixIndex(CheckIndex.java:556)
>> > at org.apache.lucene.index.CheckIndex.main(CheckIndex.java:697)
>> >
>> >
>> > Regards
>> >
>> > GC
>> >
>> >
>> >
>> > On Tue, May 26, 2009 at 12:49 AM, Peter Wolanin <
>> peter.wolanin@acquia.com>wrote:
>> >
>> >> you can use the lucene jar with solr to invoke the CheckIndex method -
>> >> this will possibly allow you to recover if you pass the -fix param.
>> >>
>> >> You may lose some docs, however, so this is only viable if you can,
>> >> for example, query to check what's missing.
>> >>
>> >> The command looks like (from the root of the solr svn checkout):
>> >>
>> >> java -ea:org.apache.lucene -cp lib/lucene-core-2.9-dev.jar
>> >> org.apache.lucene.index.CheckIndex [path to index directory]
>> >>
>> >> For example, to check the example index:
>> >>
>> >> java -ea:org.apache.lucene -cp lib/lucene-core-2.9-dev.jar
>> >> org.apache.lucene.index.CheckIndex example/solr/data/index/
>> >>
>> >> -Peter
>> >>
>> >> On Mon, May 25, 2009 at 4:42 AM, Wang Guangchen <guangchen111@gmail.com
>> >
>> >> wrote:
>> >> > Hi everyone,
>> >> >
>> >> > I have 8m docs to index, and each doc is around 50kb. The solr crashed
>> in
>> >> > the middle of indexing. error message said that one of the file in the
>> >> data
>> >> > directory is missing. I don't know why this is happened.
>> >> >
>> >> > So right now I have to find a way to recover the index to avoid
>> re-index.
>> >> Is
>> >> > there anyone know any tools or method to recover the crashed index?
>> >> Please
>> >> > help.
>> >> >
>> >> > Thanks a lot.
>> >> >
>> >> > Regards
>> >> > GC
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >> Peter M. Wolanin, Ph.D.
>> >> Momentum Specialist, Acquia. Inc.
>> >> peter.wolanin@acquia.com
>> >>
>> >
>>
>
Re: Recover crashed solr index
Posted by Wang Guangchen <gu...@gmail.com>.
Hi Mike,
The index is autoCommit every 1000 docs. I set this to increase the indexing
speed. What is the best configuration do you suggest for each commit cycle?
Thank you very much for your help.
Following is the original exception:
java.lang.RuntimeException: java.io.FileNotFoundException:
/solr/example/data/index/segments_cje (No such file or directory)
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1085)
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:561)
at
org.apache.solr.core.CoreContainer$Initializer.initialize(CoreContainer.java:121)
at
org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:69)
at
org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:275)
at
org.apache.catalina.core.ApplicationFilterConfig.setFilterDef(ApplicationFilterConfig.java:397)
at
org.apache.catalina.core.ApplicationFilterConfig.<init>(ApplicationFilterConfig.java:108)
at
org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:3709)
at
org.apache.catalina.core.StandardContext.start(StandardContext.java:4363)
at
org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:791)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:771)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:525)
at
org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:627)
at
org.apache.catalina.startup.HostConfig.deployDescriptors(HostConfig.java:553)
at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:488)
at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1149)
at
org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:311)
at
org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:117)
at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1053)
at org.apache.catalina.core.StandardHost.start(StandardHost.java:719)
at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1045)
at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:443)
at org.apache.catalina.core.StandardService.start(StandardService.java:516)
at org.apache.catalina.core.StandardServer.start(StandardServer.java:710)
at org.apache.catalina.startup.Catalina.start(Catalina.java:578)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413) Caused by:
java.io.FileNotFoundException:
/mnt_APS/solr/solrHomeFull/data/index/segments_cje (No such file or
directory)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
at
org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor.<init>(FSDirectory.java:630)
at
org.apache.lucene.store.FSDirectory$FSIndexInput.<init>(FSDirectory.java:660)
at
org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.<init>(NIOFSDirectory.java:76)
at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:63)
at org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:560)
at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:224)
at
org.apache.lucene.index.DirectoryIndexReader$1.doBody(DirectoryIndexReader.java:103)
at
org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:688)
at
org.apache.lucene.index.DirectoryIndexReader.open(DirectoryIndexReader.java:123)
at org.apache.lucene.index.IndexReader.open(IndexReader.java:316)
at org.apache.lucene.index.IndexReader.open(IndexReader.java:237)
at org.apache.solr.search.SolrIndexSearcher.<init>
On Wed, May 27, 2009 at 2:32 AM, Michael McCandless <
lucene@mikemccandless.com> wrote:
> It sounds like you had never committed while building the original
> index? Unfortunately, it's not easy to recover an index in that
> state. It's best to periodically commit if you're building such a
> large index.
>
> Do you have the original exception you hit?
>
> I'll fix CheckIndex to be more sane if it could not load any segments file.
>
> Mike
>
> On Tue, May 26, 2009 at 2:12 AM, Wang Guangchen <gu...@gmail.com>
> wrote:
> > Hi peter,
> >
> > Thank you very much for your quick reply.
> >
> > I tried the CheckIndex method. It can't work on my crashed index.
> > In the error message, it says the segments file in the directory is
> missing.
> > and when I use the -fix param, new segments file still can't be write.
> > I even try the CheckIndex without the assertion, it still can't work.
> >
> >
> > Do you know why this is happening ? Does it mean that the segment file
> can't
> > be rewrite at all?
> >
> > Btw, i am using the nightly build solr.
> >
> > following is the error messages:
> >
> > [root@localhost lib]# java -cp lucene-core-2.9-dev.jar
> > org.apache.lucene.index.CheckIndex -fix /solr/example/data/index/
> >
> > NOTE: testing will be more thorough if you run java with
> > '-ea:org.apache.lucene...', so assertions are enabled
> >
> > Opening index @ /solr/example/data/index/
> >
> > ERROR: could not read any segments file in directory
> > java.io.FileNotFoundException: /solr/example
> >
> > /data/index/segments_cje (No such file or directory)
> > at java.io.RandomAccessFile.open(Native Method)
> > at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
> > at
> >
> org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor.<init>(FSDirectory.java:630)
> > at
> >
> org.apache.lucene.store.FSDirectory$FSIndexInput.<init>(FSDirectory.java:660)
> > at
> > org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:566)
> > at
> > org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:560)
> > at
> org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:224)
> > at
> > org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:292)
> > at
> >
> org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:688)
> > at
> org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:289)
> > at
> > org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:258)
> > at org.apache.lucene.index.CheckIndex.main(CheckIndex.java:678)
> > WARNING: 0 documents will be lost
> >
> > NOTE: will write new segments file in 5 seconds; this will remove 0 docs
> > from the index. THIS IS YOUR LAST CHANCE TO CTRL+C!
> > 5...
> > 4...
> > 3...
> > 2...
> > 1...
> > Writing...
> > Exception in thread "main" java.lang.NullPointerException
> > at
> org.apache.lucene.index.CheckIndex.fixIndex(CheckIndex.java:556)
> > at org.apache.lucene.index.CheckIndex.main(CheckIndex.java:697)
> >
> >
> > Regards
> >
> > GC
> >
> >
> >
> > On Tue, May 26, 2009 at 12:49 AM, Peter Wolanin <
> peter.wolanin@acquia.com>wrote:
> >
> >> you can use the lucene jar with solr to invoke the CheckIndex method -
> >> this will possibly allow you to recover if you pass the -fix param.
> >>
> >> You may lose some docs, however, so this is only viable if you can,
> >> for example, query to check what's missing.
> >>
> >> The command looks like (from the root of the solr svn checkout):
> >>
> >> java -ea:org.apache.lucene -cp lib/lucene-core-2.9-dev.jar
> >> org.apache.lucene.index.CheckIndex [path to index directory]
> >>
> >> For example, to check the example index:
> >>
> >> java -ea:org.apache.lucene -cp lib/lucene-core-2.9-dev.jar
> >> org.apache.lucene.index.CheckIndex example/solr/data/index/
> >>
> >> -Peter
> >>
> >> On Mon, May 25, 2009 at 4:42 AM, Wang Guangchen <guangchen111@gmail.com
> >
> >> wrote:
> >> > Hi everyone,
> >> >
> >> > I have 8m docs to index, and each doc is around 50kb. The solr crashed
> in
> >> > the middle of indexing. error message said that one of the file in the
> >> data
> >> > directory is missing. I don't know why this is happened.
> >> >
> >> > So right now I have to find a way to recover the index to avoid
> re-index.
> >> Is
> >> > there anyone know any tools or method to recover the crashed index?
> >> Please
> >> > help.
> >> >
> >> > Thanks a lot.
> >> >
> >> > Regards
> >> > GC
> >> >
> >>
> >>
> >>
> >> --
> >> Peter M. Wolanin, Ph.D.
> >> Momentum Specialist, Acquia. Inc.
> >> peter.wolanin@acquia.com
> >>
> >
>
Re: Recover crashed solr index
Posted by Michael McCandless <lu...@mikemccandless.com>.
It sounds like you had never committed while building the original
index? Unfortunately, it's not easy to recover an index in that
state. It's best to periodically commit if you're building such a
large index.
Do you have the original exception you hit?
I'll fix CheckIndex to be more sane if it could not load any segments file.
Mike
On Tue, May 26, 2009 at 2:12 AM, Wang Guangchen <gu...@gmail.com> wrote:
> Hi peter,
>
> Thank you very much for your quick reply.
>
> I tried the CheckIndex method. It can't work on my crashed index.
> In the error message, it says the segments file in the directory is missing.
> and when I use the -fix param, new segments file still can't be write.
> I even try the CheckIndex without the assertion, it still can't work.
>
>
> Do you know why this is happening ? Does it mean that the segment file can't
> be rewrite at all?
>
> Btw, i am using the nightly build solr.
>
> following is the error messages:
>
> [root@localhost lib]# java -cp lucene-core-2.9-dev.jar
> org.apache.lucene.index.CheckIndex -fix /solr/example/data/index/
>
> NOTE: testing will be more thorough if you run java with
> '-ea:org.apache.lucene...', so assertions are enabled
>
> Opening index @ /solr/example/data/index/
>
> ERROR: could not read any segments file in directory
> java.io.FileNotFoundException: /solr/example
>
> /data/index/segments_cje (No such file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
> at
> org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor.<init>(FSDirectory.java:630)
> at
> org.apache.lucene.store.FSDirectory$FSIndexInput.<init>(FSDirectory.java:660)
> at
> org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:566)
> at
> org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:560)
> at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:224)
> at
> org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:292)
> at
> org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:688)
> at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:289)
> at
> org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:258)
> at org.apache.lucene.index.CheckIndex.main(CheckIndex.java:678)
> WARNING: 0 documents will be lost
>
> NOTE: will write new segments file in 5 seconds; this will remove 0 docs
> from the index. THIS IS YOUR LAST CHANCE TO CTRL+C!
> 5...
> 4...
> 3...
> 2...
> 1...
> Writing...
> Exception in thread "main" java.lang.NullPointerException
> at org.apache.lucene.index.CheckIndex.fixIndex(CheckIndex.java:556)
> at org.apache.lucene.index.CheckIndex.main(CheckIndex.java:697)
>
>
> Regards
>
> GC
>
>
>
> On Tue, May 26, 2009 at 12:49 AM, Peter Wolanin <pe...@acquia.com>wrote:
>
>> you can use the lucene jar with solr to invoke the CheckIndex method -
>> this will possibly allow you to recover if you pass the -fix param.
>>
>> You may lose some docs, however, so this is only viable if you can,
>> for example, query to check what's missing.
>>
>> The command looks like (from the root of the solr svn checkout):
>>
>> java -ea:org.apache.lucene -cp lib/lucene-core-2.9-dev.jar
>> org.apache.lucene.index.CheckIndex [path to index directory]
>>
>> For example, to check the example index:
>>
>> java -ea:org.apache.lucene -cp lib/lucene-core-2.9-dev.jar
>> org.apache.lucene.index.CheckIndex example/solr/data/index/
>>
>> -Peter
>>
>> On Mon, May 25, 2009 at 4:42 AM, Wang Guangchen <gu...@gmail.com>
>> wrote:
>> > Hi everyone,
>> >
>> > I have 8m docs to index, and each doc is around 50kb. The solr crashed in
>> > the middle of indexing. error message said that one of the file in the
>> data
>> > directory is missing. I don't know why this is happened.
>> >
>> > So right now I have to find a way to recover the index to avoid re-index.
>> Is
>> > there anyone know any tools or method to recover the crashed index?
>> Please
>> > help.
>> >
>> > Thanks a lot.
>> >
>> > Regards
>> > GC
>> >
>>
>>
>>
>> --
>> Peter M. Wolanin, Ph.D.
>> Momentum Specialist, Acquia. Inc.
>> peter.wolanin@acquia.com
>>
>
Re: Recover crashed solr index
Posted by Wang Guangchen <gu...@gmail.com>.
Hi peter,
Thank you very much for your quick reply.
I tried the CheckIndex method. It can't work on my crashed index.
In the error message, it says the segments file in the directory is missing.
and when I use the -fix param, new segments file still can't be write.
I even try the CheckIndex without the assertion, it still can't work.
Do you know why this is happening ? Does it mean that the segment file can't
be rewrite at all?
Btw, i am using the nightly build solr.
following is the error messages:
[root@localhost lib]# java -cp lucene-core-2.9-dev.jar
org.apache.lucene.index.CheckIndex -fix /solr/example/data/index/
NOTE: testing will be more thorough if you run java with
'-ea:org.apache.lucene...', so assertions are enabled
Opening index @ /solr/example/data/index/
ERROR: could not read any segments file in directory
java.io.FileNotFoundException: /solr/example
/data/index/segments_cje (No such file or directory)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
at
org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor.<init>(FSDirectory.java:630)
at
org.apache.lucene.store.FSDirectory$FSIndexInput.<init>(FSDirectory.java:660)
at
org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:566)
at
org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:560)
at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:224)
at
org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:292)
at
org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:688)
at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:289)
at
org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:258)
at org.apache.lucene.index.CheckIndex.main(CheckIndex.java:678)
WARNING: 0 documents will be lost
NOTE: will write new segments file in 5 seconds; this will remove 0 docs
from the index. THIS IS YOUR LAST CHANCE TO CTRL+C!
5...
4...
3...
2...
1...
Writing...
Exception in thread "main" java.lang.NullPointerException
at org.apache.lucene.index.CheckIndex.fixIndex(CheckIndex.java:556)
at org.apache.lucene.index.CheckIndex.main(CheckIndex.java:697)
Regards
GC
On Tue, May 26, 2009 at 12:49 AM, Peter Wolanin <pe...@acquia.com>wrote:
> you can use the lucene jar with solr to invoke the CheckIndex method -
> this will possibly allow you to recover if you pass the -fix param.
>
> You may lose some docs, however, so this is only viable if you can,
> for example, query to check what's missing.
>
> The command looks like (from the root of the solr svn checkout):
>
> java -ea:org.apache.lucene -cp lib/lucene-core-2.9-dev.jar
> org.apache.lucene.index.CheckIndex [path to index directory]
>
> For example, to check the example index:
>
> java -ea:org.apache.lucene -cp lib/lucene-core-2.9-dev.jar
> org.apache.lucene.index.CheckIndex example/solr/data/index/
>
> -Peter
>
> On Mon, May 25, 2009 at 4:42 AM, Wang Guangchen <gu...@gmail.com>
> wrote:
> > Hi everyone,
> >
> > I have 8m docs to index, and each doc is around 50kb. The solr crashed in
> > the middle of indexing. error message said that one of the file in the
> data
> > directory is missing. I don't know why this is happened.
> >
> > So right now I have to find a way to recover the index to avoid re-index.
> Is
> > there anyone know any tools or method to recover the crashed index?
> Please
> > help.
> >
> > Thanks a lot.
> >
> > Regards
> > GC
> >
>
>
>
> --
> Peter M. Wolanin, Ph.D.
> Momentum Specialist, Acquia. Inc.
> peter.wolanin@acquia.com
>
Re: Recover crashed solr index
Posted by Peter Wolanin <pe...@acquia.com>.
you can use the lucene jar with solr to invoke the CheckIndex method -
this will possibly allow you to recover if you pass the -fix param.
You may lose some docs, however, so this is only viable if you can,
for example, query to check what's missing.
The command looks like (from the root of the solr svn checkout):
java -ea:org.apache.lucene -cp lib/lucene-core-2.9-dev.jar
org.apache.lucene.index.CheckIndex [path to index directory]
For example, to check the example index:
java -ea:org.apache.lucene -cp lib/lucene-core-2.9-dev.jar
org.apache.lucene.index.CheckIndex example/solr/data/index/
-Peter
On Mon, May 25, 2009 at 4:42 AM, Wang Guangchen <gu...@gmail.com> wrote:
> Hi everyone,
>
> I have 8m docs to index, and each doc is around 50kb. The solr crashed in
> the middle of indexing. error message said that one of the file in the data
> directory is missing. I don't know why this is happened.
>
> So right now I have to find a way to recover the index to avoid re-index. Is
> there anyone know any tools or method to recover the crashed index? Please
> help.
>
> Thanks a lot.
>
> Regards
> GC
>
--
Peter M. Wolanin, Ph.D.
Momentum Specialist, Acquia. Inc.
peter.wolanin@acquia.com