You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Susheel Kumar <su...@gmail.com> on 2018/09/06 16:04:40 UTC
Corrupt Index error on Target cluster
Hello,
We had a running cluster with CDCR and there were some issues with indexing
on Source cluster which got resolved after restarting the nodes (in my
absence...) and now I see below errors on a shard at Target cluster. Any
suggestions / ideas what could have caused this and whats the best way to
recover.
Thnx
Caused by: org.apache.solr.common.SolrException: Error opening new searcher
at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:2069)
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:2189)
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1926)
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1826)
at
org.apache.solr.request.SolrQueryRequestBase.getSearcher(SolrQueryRequestBase.java:127)
at
org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:310)
at
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:296)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:173)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2477)
at
org.apache.solr.handler.PingRequestHandler.handlePing(PingRequestHandler.java:267)
... 34 more
Caused by: org.apache.lucene.index.CorruptIndexException: Corrupted
bitsPerDocBase: 6033
(resource=BufferedChecksumIndexInput(MMapIndexInput(path="/app/solr/data/COLL_shard8_replica1/data/index.20180903220548447/_9nsy.tvx")))
at
org.apache.lucene.codecs.compressing.CompressingStoredFieldsIndexReader.<init>(CompressingStoredFieldsIndexReader.java:89)
at
org.apache.lucene.codecs.compressing.CompressingTermVectorsReader.<init>(CompressingTermVectorsReader.java:126)
at
org.apache.lucene.codecs.compressing.CompressingTermVectorsFormat.vectorsReader(CompressingTermVectorsFormat.java:91)
at
org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:128)
at
org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:74)
at
org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:145)
at
org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:197)
at
org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:103)
at
org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:467)
at
org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:103)
at
org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:79)
at
org.apache.solr.core.StandardIndexReaderFactory.newReader(StandardIndexReaderFactory.java:39)
at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:2033)
... 43 more
Suppressed: org.apache.lucene.index.CorruptIndexException: checksum
failed (hardware problem?) : expected=e5bf0d15 actual=21722825
(resource=BufferedChecksumIndexInput(MMapIndexInput(path="/app/solr/data/COLL_shard8_replica1/data/index.20180903220548447/_9nsy.tvx")))
at
org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:419)
at
org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:462)
at
org.apache.lucene.codecs.compressing.CompressingTermVectorsReader.<init>(CompressingTermVectorsReader.java:131)
... 54 more
Re: Corrupt Index error on Target cluster
Posted by Stephen Bianamara <sd...@gmail.com>.
Oh, I see I missed that fix. The versions were 6.2.1 and 6.6.2 (I
mistakenly said before they were both 6.6).
On Sun, Sep 9, 2018, 2:35 AM Susheel Kumar <su...@gmail.com> wrote:
> Thanks. I have 6.6.2. Do you remember the exact minor version which you
> run into with corruptIndex. I did fix it using CheckIndex.
>
> On Sat, Sep 8, 2018 at 2:00 AM Stephen Bianamara <sdl1tinsoldier@gmail.com
> >
> wrote:
>
> > Hmm, when this occurred for me I was also on 6.6 between minor releases.
> So
> > unclear if it's connected to 6.6 specifically.
> >
> > If you want to resolve the problem, you should be able to use the
> > collection api delete that node from the collection, and then re-add it
> > which will trigger resync.
> >
> >
> > On Fri, Sep 7, 2018, 10:35 AM Susheel Kumar <su...@gmail.com>
> wrote:
> >
> > > No. The solr i have is 6.6.
> > >
> > > On Fri, Sep 7, 2018 at 10:51 AM Stephen Bianamara <
> > > sdl1tinsoldier@gmail.com>
> > > wrote:
> > >
> > > > I've gotten incorrect checksums when upgrading solr versions across
> the
> > > > cluster. Or in other words, when indexing into a mixed version
> cluster.
> > > Are
> > > > you running mixed versions by chance?
> > > >
> > > > On Fri, Sep 7, 2018, 6:07 AM Susheel Kumar <su...@gmail.com>
> > > wrote:
> > > >
> > > > > Anyone has insight / have faced above errors ?
> > > > >
> > > > > On Thu, Sep 6, 2018 at 12:04 PM Susheel Kumar <
> susheel2777@gmail.com
> > >
> > > > > wrote:
> > > > >
> > > > > > Hello,
> > > > > >
> > > > > > We had a running cluster with CDCR and there were some issues
> with
> > > > > > indexing on Source cluster which got resolved after restarting
> the
> > > > nodes
> > > > > > (in my absence...) and now I see below errors on a shard at
> Target
> > > > > > cluster. Any suggestions / ideas what could have caused this and
> > > whats
> > > > > the
> > > > > > best way to recover.
> > > > > >
> > > > > > Thnx
> > > > > >
> > > > > > Caused by: org.apache.solr.common.SolrException: Error opening
> new
> > > > > searcher
> > > > > > at
> > > > > > org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:2069)
> > > > > > at
> > > > org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:2189)
> > > > > > at
> > > > org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1926)
> > > > > > at
> > > > org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1826)
> > > > > > at
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.solr.request.SolrQueryRequestBase.getSearcher(SolrQueryRequestBase.java:127)
> > > > > > at
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:310)
> > > > > > at
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:296)
> > > > > > at
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:173)
> > > > > > at
> > org.apache.solr.core.SolrCore.execute(SolrCore.java:2477)
> > > > > > at
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.solr.handler.PingRequestHandler.handlePing(PingRequestHandler.java:267)
> > > > > > ... 34 more
> > > > > > Caused by: org.apache.lucene.index.CorruptIndexException:
> Corrupted
> > > > > > bitsPerDocBase: 6033
> > > > > >
> > > > >
> > > >
> > >
> >
> (resource=BufferedChecksumIndexInput(MMapIndexInput(path="/app/solr/data/COLL_shard8_replica1/data/index.20180903220548447/_9nsy.tvx")))
> > > > > > at
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.lucene.codecs.compressing.CompressingStoredFieldsIndexReader.<init>(CompressingStoredFieldsIndexReader.java:89)
> > > > > > at
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.lucene.codecs.compressing.CompressingTermVectorsReader.<init>(CompressingTermVectorsReader.java:126)
> > > > > > at
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.lucene.codecs.compressing.CompressingTermVectorsFormat.vectorsReader(CompressingTermVectorsFormat.java:91)
> > > > > > at
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:128)
> > > > > > at
> > > > > >
> org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:74)
> > > > > > at
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:145)
> > > > > > at
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:197)
> > > > > > at
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:103)
> > > > > > at
> > > > > >
> org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:467)
> > > > > > at
> > > > > >
> > > org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:103)
> > > > > > at
> > > > > >
> > org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:79)
> > > > > > at
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.solr.core.StandardIndexReaderFactory.newReader(StandardIndexReaderFactory.java:39)
> > > > > > at
> > > > > > org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:2033)
> > > > > > ... 43 more
> > > > > > Suppressed:
> org.apache.lucene.index.CorruptIndexException:
> > > > > > checksum failed (hardware problem?) : expected=e5bf0d15
> > > actual=21722825
> > > > > >
> > > > >
> > > >
> > >
> >
> (resource=BufferedChecksumIndexInput(MMapIndexInput(path="/app/solr/data/COLL_shard8_replica1/data/index.20180903220548447/_9nsy.tvx")))
> > > > > > at
> > > > > >
> org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:419)
> > > > > > at
> > > > > >
> org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:462)
> > > > > > at
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.lucene.codecs.compressing.CompressingTermVectorsReader.<init>(CompressingTermVectorsReader.java:131)
> > > > > > ... 54 more
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Corrupt Index error on Target cluster
Posted by Susheel Kumar <su...@gmail.com>.
Thanks. I have 6.6.2. Do you remember the exact minor version which you
run into with corruptIndex. I did fix it using CheckIndex.
On Sat, Sep 8, 2018 at 2:00 AM Stephen Bianamara <sd...@gmail.com>
wrote:
> Hmm, when this occurred for me I was also on 6.6 between minor releases. So
> unclear if it's connected to 6.6 specifically.
>
> If you want to resolve the problem, you should be able to use the
> collection api delete that node from the collection, and then re-add it
> which will trigger resync.
>
>
> On Fri, Sep 7, 2018, 10:35 AM Susheel Kumar <su...@gmail.com> wrote:
>
> > No. The solr i have is 6.6.
> >
> > On Fri, Sep 7, 2018 at 10:51 AM Stephen Bianamara <
> > sdl1tinsoldier@gmail.com>
> > wrote:
> >
> > > I've gotten incorrect checksums when upgrading solr versions across the
> > > cluster. Or in other words, when indexing into a mixed version cluster.
> > Are
> > > you running mixed versions by chance?
> > >
> > > On Fri, Sep 7, 2018, 6:07 AM Susheel Kumar <su...@gmail.com>
> > wrote:
> > >
> > > > Anyone has insight / have faced above errors ?
> > > >
> > > > On Thu, Sep 6, 2018 at 12:04 PM Susheel Kumar <susheel2777@gmail.com
> >
> > > > wrote:
> > > >
> > > > > Hello,
> > > > >
> > > > > We had a running cluster with CDCR and there were some issues with
> > > > > indexing on Source cluster which got resolved after restarting the
> > > nodes
> > > > > (in my absence...) and now I see below errors on a shard at Target
> > > > > cluster. Any suggestions / ideas what could have caused this and
> > whats
> > > > the
> > > > > best way to recover.
> > > > >
> > > > > Thnx
> > > > >
> > > > > Caused by: org.apache.solr.common.SolrException: Error opening new
> > > > searcher
> > > > > at
> > > > > org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:2069)
> > > > > at
> > > org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:2189)
> > > > > at
> > > org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1926)
> > > > > at
> > > org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1826)
> > > > > at
> > > > >
> > > >
> > >
> >
> org.apache.solr.request.SolrQueryRequestBase.getSearcher(SolrQueryRequestBase.java:127)
> > > > > at
> > > > >
> > > >
> > >
> >
> org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:310)
> > > > > at
> > > > >
> > > >
> > >
> >
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:296)
> > > > > at
> > > > >
> > > >
> > >
> >
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:173)
> > > > > at
> org.apache.solr.core.SolrCore.execute(SolrCore.java:2477)
> > > > > at
> > > > >
> > > >
> > >
> >
> org.apache.solr.handler.PingRequestHandler.handlePing(PingRequestHandler.java:267)
> > > > > ... 34 more
> > > > > Caused by: org.apache.lucene.index.CorruptIndexException: Corrupted
> > > > > bitsPerDocBase: 6033
> > > > >
> > > >
> > >
> >
> (resource=BufferedChecksumIndexInput(MMapIndexInput(path="/app/solr/data/COLL_shard8_replica1/data/index.20180903220548447/_9nsy.tvx")))
> > > > > at
> > > > >
> > > >
> > >
> >
> org.apache.lucene.codecs.compressing.CompressingStoredFieldsIndexReader.<init>(CompressingStoredFieldsIndexReader.java:89)
> > > > > at
> > > > >
> > > >
> > >
> >
> org.apache.lucene.codecs.compressing.CompressingTermVectorsReader.<init>(CompressingTermVectorsReader.java:126)
> > > > > at
> > > > >
> > > >
> > >
> >
> org.apache.lucene.codecs.compressing.CompressingTermVectorsFormat.vectorsReader(CompressingTermVectorsFormat.java:91)
> > > > > at
> > > > >
> > > >
> > >
> >
> org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:128)
> > > > > at
> > > > > org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:74)
> > > > > at
> > > > >
> > > >
> > >
> >
> org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:145)
> > > > > at
> > > > >
> > > >
> > >
> >
> org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:197)
> > > > > at
> > > > >
> > > >
> > >
> >
> org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:103)
> > > > > at
> > > > > org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:467)
> > > > > at
> > > > >
> > org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:103)
> > > > > at
> > > > >
> org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:79)
> > > > > at
> > > > >
> > > >
> > >
> >
> org.apache.solr.core.StandardIndexReaderFactory.newReader(StandardIndexReaderFactory.java:39)
> > > > > at
> > > > > org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:2033)
> > > > > ... 43 more
> > > > > Suppressed: org.apache.lucene.index.CorruptIndexException:
> > > > > checksum failed (hardware problem?) : expected=e5bf0d15
> > actual=21722825
> > > > >
> > > >
> > >
> >
> (resource=BufferedChecksumIndexInput(MMapIndexInput(path="/app/solr/data/COLL_shard8_replica1/data/index.20180903220548447/_9nsy.tvx")))
> > > > > at
> > > > > org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:419)
> > > > > at
> > > > > org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:462)
> > > > > at
> > > > >
> > > >
> > >
> >
> org.apache.lucene.codecs.compressing.CompressingTermVectorsReader.<init>(CompressingTermVectorsReader.java:131)
> > > > > ... 54 more
> > > > >
> > > >
> > >
> >
>
Re: Corrupt Index error on Target cluster
Posted by Stephen Bianamara <sd...@gmail.com>.
Hmm, when this occurred for me I was also on 6.6 between minor releases. So
unclear if it's connected to 6.6 specifically.
If you want to resolve the problem, you should be able to use the
collection api delete that node from the collection, and then re-add it
which will trigger resync.
On Fri, Sep 7, 2018, 10:35 AM Susheel Kumar <su...@gmail.com> wrote:
> No. The solr i have is 6.6.
>
> On Fri, Sep 7, 2018 at 10:51 AM Stephen Bianamara <
> sdl1tinsoldier@gmail.com>
> wrote:
>
> > I've gotten incorrect checksums when upgrading solr versions across the
> > cluster. Or in other words, when indexing into a mixed version cluster.
> Are
> > you running mixed versions by chance?
> >
> > On Fri, Sep 7, 2018, 6:07 AM Susheel Kumar <su...@gmail.com>
> wrote:
> >
> > > Anyone has insight / have faced above errors ?
> > >
> > > On Thu, Sep 6, 2018 at 12:04 PM Susheel Kumar <su...@gmail.com>
> > > wrote:
> > >
> > > > Hello,
> > > >
> > > > We had a running cluster with CDCR and there were some issues with
> > > > indexing on Source cluster which got resolved after restarting the
> > nodes
> > > > (in my absence...) and now I see below errors on a shard at Target
> > > > cluster. Any suggestions / ideas what could have caused this and
> whats
> > > the
> > > > best way to recover.
> > > >
> > > > Thnx
> > > >
> > > > Caused by: org.apache.solr.common.SolrException: Error opening new
> > > searcher
> > > > at
> > > > org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:2069)
> > > > at
> > org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:2189)
> > > > at
> > org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1926)
> > > > at
> > org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1826)
> > > > at
> > > >
> > >
> >
> org.apache.solr.request.SolrQueryRequestBase.getSearcher(SolrQueryRequestBase.java:127)
> > > > at
> > > >
> > >
> >
> org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:310)
> > > > at
> > > >
> > >
> >
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:296)
> > > > at
> > > >
> > >
> >
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:173)
> > > > at org.apache.solr.core.SolrCore.execute(SolrCore.java:2477)
> > > > at
> > > >
> > >
> >
> org.apache.solr.handler.PingRequestHandler.handlePing(PingRequestHandler.java:267)
> > > > ... 34 more
> > > > Caused by: org.apache.lucene.index.CorruptIndexException: Corrupted
> > > > bitsPerDocBase: 6033
> > > >
> > >
> >
> (resource=BufferedChecksumIndexInput(MMapIndexInput(path="/app/solr/data/COLL_shard8_replica1/data/index.20180903220548447/_9nsy.tvx")))
> > > > at
> > > >
> > >
> >
> org.apache.lucene.codecs.compressing.CompressingStoredFieldsIndexReader.<init>(CompressingStoredFieldsIndexReader.java:89)
> > > > at
> > > >
> > >
> >
> org.apache.lucene.codecs.compressing.CompressingTermVectorsReader.<init>(CompressingTermVectorsReader.java:126)
> > > > at
> > > >
> > >
> >
> org.apache.lucene.codecs.compressing.CompressingTermVectorsFormat.vectorsReader(CompressingTermVectorsFormat.java:91)
> > > > at
> > > >
> > >
> >
> org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:128)
> > > > at
> > > > org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:74)
> > > > at
> > > >
> > >
> >
> org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:145)
> > > > at
> > > >
> > >
> >
> org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:197)
> > > > at
> > > >
> > >
> >
> org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:103)
> > > > at
> > > > org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:467)
> > > > at
> > > >
> org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:103)
> > > > at
> > > > org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:79)
> > > > at
> > > >
> > >
> >
> org.apache.solr.core.StandardIndexReaderFactory.newReader(StandardIndexReaderFactory.java:39)
> > > > at
> > > > org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:2033)
> > > > ... 43 more
> > > > Suppressed: org.apache.lucene.index.CorruptIndexException:
> > > > checksum failed (hardware problem?) : expected=e5bf0d15
> actual=21722825
> > > >
> > >
> >
> (resource=BufferedChecksumIndexInput(MMapIndexInput(path="/app/solr/data/COLL_shard8_replica1/data/index.20180903220548447/_9nsy.tvx")))
> > > > at
> > > > org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:419)
> > > > at
> > > > org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:462)
> > > > at
> > > >
> > >
> >
> org.apache.lucene.codecs.compressing.CompressingTermVectorsReader.<init>(CompressingTermVectorsReader.java:131)
> > > > ... 54 more
> > > >
> > >
> >
>
Re: Corrupt Index error on Target cluster
Posted by Susheel Kumar <su...@gmail.com>.
No. The solr i have is 6.6.
On Fri, Sep 7, 2018 at 10:51 AM Stephen Bianamara <sd...@gmail.com>
wrote:
> I've gotten incorrect checksums when upgrading solr versions across the
> cluster. Or in other words, when indexing into a mixed version cluster. Are
> you running mixed versions by chance?
>
> On Fri, Sep 7, 2018, 6:07 AM Susheel Kumar <su...@gmail.com> wrote:
>
> > Anyone has insight / have faced above errors ?
> >
> > On Thu, Sep 6, 2018 at 12:04 PM Susheel Kumar <su...@gmail.com>
> > wrote:
> >
> > > Hello,
> > >
> > > We had a running cluster with CDCR and there were some issues with
> > > indexing on Source cluster which got resolved after restarting the
> nodes
> > > (in my absence...) and now I see below errors on a shard at Target
> > > cluster. Any suggestions / ideas what could have caused this and whats
> > the
> > > best way to recover.
> > >
> > > Thnx
> > >
> > > Caused by: org.apache.solr.common.SolrException: Error opening new
> > searcher
> > > at
> > > org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:2069)
> > > at
> org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:2189)
> > > at
> org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1926)
> > > at
> org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1826)
> > > at
> > >
> >
> org.apache.solr.request.SolrQueryRequestBase.getSearcher(SolrQueryRequestBase.java:127)
> > > at
> > >
> >
> org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:310)
> > > at
> > >
> >
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:296)
> > > at
> > >
> >
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:173)
> > > at org.apache.solr.core.SolrCore.execute(SolrCore.java:2477)
> > > at
> > >
> >
> org.apache.solr.handler.PingRequestHandler.handlePing(PingRequestHandler.java:267)
> > > ... 34 more
> > > Caused by: org.apache.lucene.index.CorruptIndexException: Corrupted
> > > bitsPerDocBase: 6033
> > >
> >
> (resource=BufferedChecksumIndexInput(MMapIndexInput(path="/app/solr/data/COLL_shard8_replica1/data/index.20180903220548447/_9nsy.tvx")))
> > > at
> > >
> >
> org.apache.lucene.codecs.compressing.CompressingStoredFieldsIndexReader.<init>(CompressingStoredFieldsIndexReader.java:89)
> > > at
> > >
> >
> org.apache.lucene.codecs.compressing.CompressingTermVectorsReader.<init>(CompressingTermVectorsReader.java:126)
> > > at
> > >
> >
> org.apache.lucene.codecs.compressing.CompressingTermVectorsFormat.vectorsReader(CompressingTermVectorsFormat.java:91)
> > > at
> > >
> >
> org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:128)
> > > at
> > > org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:74)
> > > at
> > >
> >
> org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:145)
> > > at
> > >
> >
> org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:197)
> > > at
> > >
> >
> org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:103)
> > > at
> > > org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:467)
> > > at
> > > org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:103)
> > > at
> > > org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:79)
> > > at
> > >
> >
> org.apache.solr.core.StandardIndexReaderFactory.newReader(StandardIndexReaderFactory.java:39)
> > > at
> > > org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:2033)
> > > ... 43 more
> > > Suppressed: org.apache.lucene.index.CorruptIndexException:
> > > checksum failed (hardware problem?) : expected=e5bf0d15 actual=21722825
> > >
> >
> (resource=BufferedChecksumIndexInput(MMapIndexInput(path="/app/solr/data/COLL_shard8_replica1/data/index.20180903220548447/_9nsy.tvx")))
> > > at
> > > org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:419)
> > > at
> > > org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:462)
> > > at
> > >
> >
> org.apache.lucene.codecs.compressing.CompressingTermVectorsReader.<init>(CompressingTermVectorsReader.java:131)
> > > ... 54 more
> > >
> >
>
Re: Corrupt Index error on Target cluster
Posted by Stephen Bianamara <sd...@gmail.com>.
I've gotten incorrect checksums when upgrading solr versions across the
cluster. Or in other words, when indexing into a mixed version cluster. Are
you running mixed versions by chance?
On Fri, Sep 7, 2018, 6:07 AM Susheel Kumar <su...@gmail.com> wrote:
> Anyone has insight / have faced above errors ?
>
> On Thu, Sep 6, 2018 at 12:04 PM Susheel Kumar <su...@gmail.com>
> wrote:
>
> > Hello,
> >
> > We had a running cluster with CDCR and there were some issues with
> > indexing on Source cluster which got resolved after restarting the nodes
> > (in my absence...) and now I see below errors on a shard at Target
> > cluster. Any suggestions / ideas what could have caused this and whats
> the
> > best way to recover.
> >
> > Thnx
> >
> > Caused by: org.apache.solr.common.SolrException: Error opening new
> searcher
> > at
> > org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:2069)
> > at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:2189)
> > at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1926)
> > at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1826)
> > at
> >
> org.apache.solr.request.SolrQueryRequestBase.getSearcher(SolrQueryRequestBase.java:127)
> > at
> >
> org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:310)
> > at
> >
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:296)
> > at
> >
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:173)
> > at org.apache.solr.core.SolrCore.execute(SolrCore.java:2477)
> > at
> >
> org.apache.solr.handler.PingRequestHandler.handlePing(PingRequestHandler.java:267)
> > ... 34 more
> > Caused by: org.apache.lucene.index.CorruptIndexException: Corrupted
> > bitsPerDocBase: 6033
> >
> (resource=BufferedChecksumIndexInput(MMapIndexInput(path="/app/solr/data/COLL_shard8_replica1/data/index.20180903220548447/_9nsy.tvx")))
> > at
> >
> org.apache.lucene.codecs.compressing.CompressingStoredFieldsIndexReader.<init>(CompressingStoredFieldsIndexReader.java:89)
> > at
> >
> org.apache.lucene.codecs.compressing.CompressingTermVectorsReader.<init>(CompressingTermVectorsReader.java:126)
> > at
> >
> org.apache.lucene.codecs.compressing.CompressingTermVectorsFormat.vectorsReader(CompressingTermVectorsFormat.java:91)
> > at
> >
> org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:128)
> > at
> > org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:74)
> > at
> >
> org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:145)
> > at
> >
> org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:197)
> > at
> >
> org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:103)
> > at
> > org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:467)
> > at
> > org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:103)
> > at
> > org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:79)
> > at
> >
> org.apache.solr.core.StandardIndexReaderFactory.newReader(StandardIndexReaderFactory.java:39)
> > at
> > org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:2033)
> > ... 43 more
> > Suppressed: org.apache.lucene.index.CorruptIndexException:
> > checksum failed (hardware problem?) : expected=e5bf0d15 actual=21722825
> >
> (resource=BufferedChecksumIndexInput(MMapIndexInput(path="/app/solr/data/COLL_shard8_replica1/data/index.20180903220548447/_9nsy.tvx")))
> > at
> > org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:419)
> > at
> > org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:462)
> > at
> >
> org.apache.lucene.codecs.compressing.CompressingTermVectorsReader.<init>(CompressingTermVectorsReader.java:131)
> > ... 54 more
> >
>
Re: Corrupt Index error on Target cluster
Posted by Susheel Kumar <su...@gmail.com>.
Anyone has insight / have faced above errors ?
On Thu, Sep 6, 2018 at 12:04 PM Susheel Kumar <su...@gmail.com> wrote:
> Hello,
>
> We had a running cluster with CDCR and there were some issues with
> indexing on Source cluster which got resolved after restarting the nodes
> (in my absence...) and now I see below errors on a shard at Target
> cluster. Any suggestions / ideas what could have caused this and whats the
> best way to recover.
>
> Thnx
>
> Caused by: org.apache.solr.common.SolrException: Error opening new searcher
> at
> org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:2069)
> at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:2189)
> at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1926)
> at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1826)
> at
> org.apache.solr.request.SolrQueryRequestBase.getSearcher(SolrQueryRequestBase.java:127)
> at
> org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:310)
> at
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:296)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:173)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2477)
> at
> org.apache.solr.handler.PingRequestHandler.handlePing(PingRequestHandler.java:267)
> ... 34 more
> Caused by: org.apache.lucene.index.CorruptIndexException: Corrupted
> bitsPerDocBase: 6033
> (resource=BufferedChecksumIndexInput(MMapIndexInput(path="/app/solr/data/COLL_shard8_replica1/data/index.20180903220548447/_9nsy.tvx")))
> at
> org.apache.lucene.codecs.compressing.CompressingStoredFieldsIndexReader.<init>(CompressingStoredFieldsIndexReader.java:89)
> at
> org.apache.lucene.codecs.compressing.CompressingTermVectorsReader.<init>(CompressingTermVectorsReader.java:126)
> at
> org.apache.lucene.codecs.compressing.CompressingTermVectorsFormat.vectorsReader(CompressingTermVectorsFormat.java:91)
> at
> org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:128)
> at
> org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:74)
> at
> org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:145)
> at
> org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:197)
> at
> org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:103)
> at
> org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:467)
> at
> org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:103)
> at
> org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:79)
> at
> org.apache.solr.core.StandardIndexReaderFactory.newReader(StandardIndexReaderFactory.java:39)
> at
> org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:2033)
> ... 43 more
> Suppressed: org.apache.lucene.index.CorruptIndexException:
> checksum failed (hardware problem?) : expected=e5bf0d15 actual=21722825
> (resource=BufferedChecksumIndexInput(MMapIndexInput(path="/app/solr/data/COLL_shard8_replica1/data/index.20180903220548447/_9nsy.tvx")))
> at
> org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:419)
> at
> org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:462)
> at
> org.apache.lucene.codecs.compressing.CompressingTermVectorsReader.<init>(CompressingTermVectorsReader.java:131)
> ... 54 more
>