You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "Cassandra Targett (Jira)" <ji...@apache.org> on 2020/02/10 22:58:00 UTC
[jira] [Comment Edited] (SOLR-14254) Index backcompat break between
8.3.1 and 8.4.1
[ https://issues.apache.org/jira/browse/SOLR-14254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17033993#comment-17033993 ]
Cassandra Targett edited comment on SOLR-14254 at 2/10/20 10:57 PM:
--------------------------------------------------------------------
[~dsmiley], do you have a recommendation for people who use the Tagger handler in 8.x and who upgrade to 8.4? Is the only option to reindex?
LUCENE-9116 is for 8.5, so it was a bit confusing to find the break in 8.4.1, but my reading of the comments there is that something was worked out - did I misunderstand that discussion (very possible since I didn't look at the code)?
Also, {{tagger-handler.adoc}} for 8.4 and 8.5 still says the {{postingsFormat=FST50}} is the official recommendation. That would obviously fail - should the {{postingsFormat}} param now be omitted?
Edit: re-reading Hoss' comment I think I missed that the new codec is using the same name as the old one (or maybe that's just on master after LUCENE-9116), which would mean docs do not need to be updated, but it would be nice to get explicit clarification on that. The docs still need to be updated with a major caveat that using the param we tell them to use means they are now in "non-standard codec" territory and stuff could shift without warning.
was (Author: ctargett):
[~dsmiley], do you have a recommendation for people who use the Tagger handler in 8.x and who upgrade to 8.4? Is the only option to reindex?
LUCENE-9116 is for 8.5, so it was a bit confusing to find the break in 8.4.1, but my reading of the comments there is that something was worked out - did I misunderstand that discussion (very possible since I didn't look at the code)?
Also, {{tagger-handler.adoc}} for 8.4 and 8.5 still says the {{postingsFormat=FST50}} is the official recommendation. That would obviously fail - should the {{postingsFormat}} param now be omitted?
> Index backcompat break between 8.3.1 and 8.4.1
> ----------------------------------------------
>
> Key: SOLR-14254
> URL: https://issues.apache.org/jira/browse/SOLR-14254
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Reporter: Jason Gerlowski
> Priority: Major
>
> I believe I found a backcompat break between 8.4.1 and 8.3.1.
> I encountered this when a Solr 8.3.1 cluster was upgraded to 8.4.1. On 8.4. nodes, several collections had cores fail to come up with {{CorruptIndexException}}:
> {code}
> 2020-02-10 20:58:26.136 ERROR (coreContainerWorkExecutor-2-thread-1-processing-n:192.168.1.194:8983_solr) [ ] o.a.s.c.CoreContainer Error waiting for SolrCore to be loaded on startup => org.apache.sol
> r.common.SolrException: Unable to create core [testbackcompat_shard1_replica_n1]
> at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1313)
> org.apache.solr.common.SolrException: Unable to create core [testbackcompat_shard1_replica_n1]
> at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1313) ~[?:?]
> at org.apache.solr.core.CoreContainer.lambda$load$13(CoreContainer.java:788) ~[?:?]
> at com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:202) ~[metrics-core-4.0.5.jar:4.0.5]
> at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:210) ~[?:?]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
> at java.lang.Thread.run(Thread.java:834) [?:?]
> Caused by: org.apache.solr.common.SolrException: Error opening new searcher
> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1072) ~[?:?]
> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:901) ~[?:?]
> at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1292) ~[?:?]
> ... 7 more
> Caused by: org.apache.solr.common.SolrException: Error opening new searcher
> at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:2182) ~[?:?]
> at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:2302) ~[?:?]
> at org.apache.solr.core.SolrCore.initSearcher(SolrCore.java:1132) ~[?:?]
> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1013) ~[?:?]
> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:901) ~[?:?]
> at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1292) ~[?:?]
> ... 7 more
> Caused by: org.apache.lucene.index.CorruptIndexException: codec mismatch: actual codec=Lucene50PostingsWriterDoc vs expected codec=Lucene84PostingsWriterDoc (resource=MMapIndexInput(path="/Users/jasongerlowski/run/solrdata/data/testbackcompat_shard1_replica_n1/data/index/_0_FST50_0.doc"))
> at org.apache.lucene.codecs.CodecUtil.checkHeaderNoMagic(CodecUtil.java:208) ~[?:?]
> at org.apache.lucene.codecs.CodecUtil.checkHeader(CodecUtil.java:198) ~[?:?]
> at org.apache.lucene.codecs.CodecUtil.checkIndexHeader(CodecUtil.java:255) ~[?:?]
> at org.apache.lucene.codecs.lucene84.Lucene84PostingsReader.<init>(Lucene84PostingsReader.java:82) ~[?:?]
> at org.apache.lucene.codecs.memory.FSTPostingsFormat.fieldsProducer(FSTPostingsFormat.java:66) ~[?:?]
> at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.<init>(PerFieldPostingsFormat.java:315) ~[?:?]
> at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat.fieldsProducer(PerFieldPostingsFormat.java:395) ~[?:?]
> at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:114) ~[?:?]
> at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:84) ~[?:?]
> at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:177) ~[?:?]
> at org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:219) ~[?:?]
> at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:109) ~[?:?]
> at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:526) ~[?:?]
> at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:116) ~[?:?]
> at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:92) ~[?:?]
> at org.apache.solr.core.StandardIndexReaderFactory.newReader(StandardIndexReaderFactory.java:39) ~[?:?]
> at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:2146) ~[?:?]
> at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:2302) ~[?:?]
> at org.apache.solr.core.SolrCore.initSearcher(SolrCore.java:1132) ~[?:?]
> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1013) ~[?:?]
> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:901) ~[?:?]
> at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1292) ~[?:?]
> ... 7 more
> {code}
> Only some cores were affected. After some digging, we noticed that these cores were unique in that they had a field set up to use Solr's Text Tagger. Not the use of a non-default postings-format (FST50).
> {code}
> <fieldType class="solr.TextField" name="tagger" omitNorms="true" omitTermFreqAndPositions="true" postingsFormat="FST50">
> <analyzer type="index">
> <tokenizer class="solr.StandardTokenizerFactory"/>
> <filter class="solr.EnglishPossessiveFilterFactory"/>
> <filter class="solr.ASCIIFoldingFilterFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> <filter class="solr.ConcatenateGraphFilterFactory" preservePositionIncrements="false"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.StandardTokenizerFactory"/>
> <filter class="solr.EnglishPossessiveFilterFactory"/>
> <filter class="solr.ASCIIFoldingFilterFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
> <field name="tagger_field" type="tagger" indexed="true" stored="true" />
> {code}
> I've since been able to reproduce the problem locally with the following steps:
> # Start Solr 8.3.1 with a data directory /some/data/dir
> # Modify the default configset (or create a new configset) by adding the fieldType/field above to your schema.
> # Index a document that contains a "tagger_field" value and commit.
> # Stop Solr 8.3.1
> # Start Solr 8.4.1 pointed at the /some/data/dir as its data directory.
> At this point, the core should be down and you should see the CorruptIndexException in your logs.
> This _seems_ related to LUCENE-9027, though I don't know enough about how Lucene handles ensuring support for older indices to know exactly what happened there.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org