You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "Jason Gerlowski (Jira)" <ji...@apache.org> on 2020/02/10 21:39:00 UTC
[jira] [Updated] (SOLR-14254) Index backcompat break between 8.3.1
and 8.4.1
[ https://issues.apache.org/jira/browse/SOLR-14254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jason Gerlowski updated SOLR-14254:
-----------------------------------
Description:
I believe I found a backcompat break between 8.4.1 and 8.3.1.
I encountered this when a Solr 8.3.1 cluster was upgraded to 8.4.1. On 8.4. nodes, several collections had cores fail to come up with {{CorruptIndexException}}:
{code}
2020-02-10 20:58:26.136 ERROR (coreContainerWorkExecutor-2-thread-1-processing-n:192.168.1.194:8983_solr) [ ] o.a.s.c.CoreContainer Error waiting for SolrCore to be loaded on startup => org.apache.sol
r.common.SolrException: Unable to create core [testbackcompat_shard1_replica_n1]
at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1313)
org.apache.solr.common.SolrException: Unable to create core [testbackcompat_shard1_replica_n1]
at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1313) ~[?:?]
at org.apache.solr.core.CoreContainer.lambda$load$13(CoreContainer.java:788) ~[?:?]
at com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:202) ~[metrics-core-4.0.5.jar:4.0.5]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:210) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.apache.solr.common.SolrException: Error opening new searcher
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1072) ~[?:?]
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:901) ~[?:?]
at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1292) ~[?:?]
... 7 more
Caused by: org.apache.solr.common.SolrException: Error opening new searcher
at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:2182) ~[?:?]
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:2302) ~[?:?]
at org.apache.solr.core.SolrCore.initSearcher(SolrCore.java:1132) ~[?:?]
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1013) ~[?:?]
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:901) ~[?:?]
at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1292) ~[?:?]
... 7 more
Caused by: org.apache.lucene.index.CorruptIndexException: codec mismatch: actual codec=Lucene50PostingsWriterDoc vs expected codec=Lucene84PostingsWriterDoc (resource=MMapIndexInput(path="/Users/jasongerlowski/run/solrdata/data/testbackcompat_shard1_replica_n1/data/index/_0_FST50_0.doc"))
at org.apache.lucene.codecs.CodecUtil.checkHeaderNoMagic(CodecUtil.java:208) ~[?:?]
at org.apache.lucene.codecs.CodecUtil.checkHeader(CodecUtil.java:198) ~[?:?]
at org.apache.lucene.codecs.CodecUtil.checkIndexHeader(CodecUtil.java:255) ~[?:?]
at org.apache.lucene.codecs.lucene84.Lucene84PostingsReader.<init>(Lucene84PostingsReader.java:82) ~[?:?]
at org.apache.lucene.codecs.memory.FSTPostingsFormat.fieldsProducer(FSTPostingsFormat.java:66) ~[?:?]
at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.<init>(PerFieldPostingsFormat.java:315) ~[?:?]
at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat.fieldsProducer(PerFieldPostingsFormat.java:395) ~[?:?]
at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:114) ~[?:?]
at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:84) ~[?:?]
at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:177) ~[?:?]
at org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:219) ~[?:?]
at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:109) ~[?:?]
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:526) ~[?:?]
at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:116) ~[?:?]
at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:92) ~[?:?]
at org.apache.solr.core.StandardIndexReaderFactory.newReader(StandardIndexReaderFactory.java:39) ~[?:?]
at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:2146) ~[?:?]
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:2302) ~[?:?]
at org.apache.solr.core.SolrCore.initSearcher(SolrCore.java:1132) ~[?:?]
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1013) ~[?:?]
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:901) ~[?:?]
at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1292) ~[?:?]
... 7 more
{code}
Only some cores were affected. After some digging, we noticed that these cores were unique in that they had a field set up to use Solr's Text Tagger. Not the use of a non-default postings-format (FST50).
{code}
<fieldType class="solr.TextField" name="tagger" omitNorms="true" omitTermFreqAndPositions="true" postingsFormat="FST50">
<analyzer type="index">
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.EnglishPossessiveFilterFactory"/>
<filter class="solr.ASCIIFoldingFilterFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.ConcatenateGraphFilterFactory" preservePositionIncrements="false"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.EnglishPossessiveFilterFactory"/>
<filter class="solr.ASCIIFoldingFilterFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<field name="tagger_field" type="tagger" indexed="true" stored="true" />
{code}
I've since been able to reproduce the problem locally with the following steps:
# Start Solr 8.3.1 with a data directory /some/data/dir
# Modify the default configset (or create a new configset) by adding the fieldType/field above to your schema.
# Index a document that contains a "tagger_field" value and commit.
# Stop Solr 8.3.1
# Start Solr 8.4.1 pointed at the /some/data/dir as its data directory.
At this point, the core should be down and you should see the CorruptIndexException in your logs.
This _seems_ related to LUCENE-9027, though I don't know enough about how Lucene handles ensuring support for older indices to know exactly what happened there.
was:
I believe I found a backcompat break between 8.4.1 and 8.3.1.
I encountered this when a Solr 8.3.1 cluster was upgraded to 8.4.1. On 8.4. nodes, several collections had cores fail to come up with {{CorruptIndexException}}:
{code}
2020-02-10 20:58:26.136 ERROR (coreContainerWorkExecutor-2-thread-1-processing-n:192.168.1.194:8983_solr) [ ] o.a.s.c.CoreContainer Error waiting for SolrCore to be loaded on startup => org.apache.sol
r.common.SolrException: Unable to create core [testbackcompat_shard1_replica_n1]
at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1313)
org.apache.solr.common.SolrException: Unable to create core [testbackcompat_shard1_replica_n1]
at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1313) ~[?:?]
at org.apache.solr.core.CoreContainer.lambda$load$13(CoreContainer.java:788) ~[?:?]
at com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:202) ~[metrics-core-4.0.5.jar:4.0.5]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:210) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.apache.solr.common.SolrException: Error opening new searcher
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1072) ~[?:?]
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:901) ~[?:?]
at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1292) ~[?:?]
... 7 more
Caused by: org.apache.solr.common.SolrException: Error opening new searcher
at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:2182) ~[?:?]
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:2302) ~[?:?]
at org.apache.solr.core.SolrCore.initSearcher(SolrCore.java:1132) ~[?:?]
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1013) ~[?:?]
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:901) ~[?:?]
at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1292) ~[?:?]
... 7 more
Caused by: org.apache.lucene.index.CorruptIndexException: codec mismatch: actual codec=Lucene50PostingsWriterDoc vs expected codec=Lucene84PostingsWriterDoc (resource=MMapIndexInput(path="/Users/jasongerlowski/run/solrdata/data/testbackcompat_shard1_replica_n1/data/index/_0_FST50_0.doc"))
at org.apache.lucene.codecs.CodecUtil.checkHeaderNoMagic(CodecUtil.java:208) ~[?:?]
at org.apache.lucene.codecs.CodecUtil.checkHeader(CodecUtil.java:198) ~[?:?]
at org.apache.lucene.codecs.CodecUtil.checkIndexHeader(CodecUtil.java:255) ~[?:?]
at org.apache.lucene.codecs.lucene84.Lucene84PostingsReader.<init>(Lucene84PostingsReader.java:82) ~[?:?]
at org.apache.lucene.codecs.memory.FSTPostingsFormat.fieldsProducer(FSTPostingsFormat.java:66) ~[?:?]
at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.<init>(PerFieldPostingsFormat.java:315) ~[?:?]
at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat.fieldsProducer(PerFieldPostingsFormat.java:395) ~[?:?]
at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:114) ~[?:?]
at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:84) ~[?:?]
at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:177) ~[?:?]
at org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:219) ~[?:?]
at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:109) ~[?:?]
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:526) ~[?:?]
at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:116) ~[?:?]
at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:92) ~[?:?]
at org.apache.solr.core.StandardIndexReaderFactory.newReader(StandardIndexReaderFactory.java:39) ~[?:?]
at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:2146) ~[?:?]
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:2302) ~[?:?]
at org.apache.solr.core.SolrCore.initSearcher(SolrCore.java:1132) ~[?:?]
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1013) ~[?:?]
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:901) ~[?:?]
at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1292) ~[?:?]
... 7 more
{code}
Only some cores were affected. After some digging, we noticed that these cores were unique in that they had a field set up to use Solr's Text Tagger:
{code}
<fieldType class="solr.TextField" name="tagger" omitNorms="true" omitTermFreqAndPositions="true" postingsFormat="FST50">
<analyzer type="index">
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.EnglishPossessiveFilterFactory"/>
<filter class="solr.ASCIIFoldingFilterFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.ConcatenateGraphFilterFactory" preservePositionIncrements="false"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.EnglishPossessiveFilterFactory"/>
<filter class="solr.ASCIIFoldingFilterFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<field name="tagger_field" type="tagger" indexed="true" stored="true" />
{code}
I've since been able to reproduce the problem locally with the following steps:
# Start Solr 8.3.1 with a data directory /some/data/dir
# Modify the default configset (or create a new configset) by adding the fieldType/field above to your schema.
# Index a document that contains a "tagger_field" value and commit.
# Stop Solr 8.3.1
# Start Solr 8.4.1 pointed at the /some/data/dir as its data directory.
At this point, the core should be down and you should see the CorruptIndexException in your logs.
This _seems_ related to LUCENE-9027, though I don't know enough about how Lucene handles ensuring support for older indices to know exactly what happened there.
> Index backcompat break between 8.3.1 and 8.4.1
> ----------------------------------------------
>
> Key: SOLR-14254
> URL: https://issues.apache.org/jira/browse/SOLR-14254
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Reporter: Jason Gerlowski
> Priority: Major
>
> I believe I found a backcompat break between 8.4.1 and 8.3.1.
> I encountered this when a Solr 8.3.1 cluster was upgraded to 8.4.1. On 8.4. nodes, several collections had cores fail to come up with {{CorruptIndexException}}:
> {code}
> 2020-02-10 20:58:26.136 ERROR (coreContainerWorkExecutor-2-thread-1-processing-n:192.168.1.194:8983_solr) [ ] o.a.s.c.CoreContainer Error waiting for SolrCore to be loaded on startup => org.apache.sol
> r.common.SolrException: Unable to create core [testbackcompat_shard1_replica_n1]
> at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1313)
> org.apache.solr.common.SolrException: Unable to create core [testbackcompat_shard1_replica_n1]
> at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1313) ~[?:?]
> at org.apache.solr.core.CoreContainer.lambda$load$13(CoreContainer.java:788) ~[?:?]
> at com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:202) ~[metrics-core-4.0.5.jar:4.0.5]
> at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:210) ~[?:?]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
> at java.lang.Thread.run(Thread.java:834) [?:?]
> Caused by: org.apache.solr.common.SolrException: Error opening new searcher
> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1072) ~[?:?]
> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:901) ~[?:?]
> at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1292) ~[?:?]
> ... 7 more
> Caused by: org.apache.solr.common.SolrException: Error opening new searcher
> at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:2182) ~[?:?]
> at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:2302) ~[?:?]
> at org.apache.solr.core.SolrCore.initSearcher(SolrCore.java:1132) ~[?:?]
> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1013) ~[?:?]
> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:901) ~[?:?]
> at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1292) ~[?:?]
> ... 7 more
> Caused by: org.apache.lucene.index.CorruptIndexException: codec mismatch: actual codec=Lucene50PostingsWriterDoc vs expected codec=Lucene84PostingsWriterDoc (resource=MMapIndexInput(path="/Users/jasongerlowski/run/solrdata/data/testbackcompat_shard1_replica_n1/data/index/_0_FST50_0.doc"))
> at org.apache.lucene.codecs.CodecUtil.checkHeaderNoMagic(CodecUtil.java:208) ~[?:?]
> at org.apache.lucene.codecs.CodecUtil.checkHeader(CodecUtil.java:198) ~[?:?]
> at org.apache.lucene.codecs.CodecUtil.checkIndexHeader(CodecUtil.java:255) ~[?:?]
> at org.apache.lucene.codecs.lucene84.Lucene84PostingsReader.<init>(Lucene84PostingsReader.java:82) ~[?:?]
> at org.apache.lucene.codecs.memory.FSTPostingsFormat.fieldsProducer(FSTPostingsFormat.java:66) ~[?:?]
> at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.<init>(PerFieldPostingsFormat.java:315) ~[?:?]
> at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat.fieldsProducer(PerFieldPostingsFormat.java:395) ~[?:?]
> at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:114) ~[?:?]
> at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:84) ~[?:?]
> at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:177) ~[?:?]
> at org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:219) ~[?:?]
> at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:109) ~[?:?]
> at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:526) ~[?:?]
> at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:116) ~[?:?]
> at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:92) ~[?:?]
> at org.apache.solr.core.StandardIndexReaderFactory.newReader(StandardIndexReaderFactory.java:39) ~[?:?]
> at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:2146) ~[?:?]
> at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:2302) ~[?:?]
> at org.apache.solr.core.SolrCore.initSearcher(SolrCore.java:1132) ~[?:?]
> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1013) ~[?:?]
> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:901) ~[?:?]
> at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1292) ~[?:?]
> ... 7 more
> {code}
> Only some cores were affected. After some digging, we noticed that these cores were unique in that they had a field set up to use Solr's Text Tagger. Not the use of a non-default postings-format (FST50).
> {code}
> <fieldType class="solr.TextField" name="tagger" omitNorms="true" omitTermFreqAndPositions="true" postingsFormat="FST50">
> <analyzer type="index">
> <tokenizer class="solr.StandardTokenizerFactory"/>
> <filter class="solr.EnglishPossessiveFilterFactory"/>
> <filter class="solr.ASCIIFoldingFilterFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> <filter class="solr.ConcatenateGraphFilterFactory" preservePositionIncrements="false"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.StandardTokenizerFactory"/>
> <filter class="solr.EnglishPossessiveFilterFactory"/>
> <filter class="solr.ASCIIFoldingFilterFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
> <field name="tagger_field" type="tagger" indexed="true" stored="true" />
> {code}
> I've since been able to reproduce the problem locally with the following steps:
> # Start Solr 8.3.1 with a data directory /some/data/dir
> # Modify the default configset (or create a new configset) by adding the fieldType/field above to your schema.
> # Index a document that contains a "tagger_field" value and commit.
> # Stop Solr 8.3.1
> # Start Solr 8.4.1 pointed at the /some/data/dir as its data directory.
> At this point, the core should be down and you should see the CorruptIndexException in your logs.
> This _seems_ related to LUCENE-9027, though I don't know enough about how Lucene handles ensuring support for older indices to know exactly what happened there.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org