Collections Fail to load after Solr Restart

[Webster Homer <we...@sial.com>]

Yesterday I restarted a development solrcloud. After the cloud restarted 2
collections failed to come back.

I see this in the log:
2018-02-16 15:31:16.684 ERROR
(coreLoadExecutor-6-thread-1-processing-n:ae1c-ecomdev-msc02:8983_solr) [
 ] o.a.s.c.CachingDirectoryFactory Error closing
directory:org.apache.solr.common.SolrException: Timeout waiting for all
directory ref counts to be released - gave up waiting on
CachedDir<<refCount=1;path=/var/solr/data/bb-catalog-material_shard1_replica1/data/index;done=false>>
at
org.apache.solr.core.CachingDirectoryFactory.close(CachingDirectoryFactory.java:178)
at org.apache.solr.core.SolrCore.close(SolrCore.java:1633)
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1000)
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:863)
at
org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1037)
at org.apache.solr.core.CoreContainer.lambda$load$13(CoreContainer.java:642)
at
com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:197)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)


right after this I also see the following error:
2018-02-16 15:31:16.713 ERROR
(coreContainerWorkExecutor-2-thread-1-processing-n:ae1c-ecomdev-msc02:8983_solr)
[   ] o.a.s.c.CoreContainer Error waiting for SolrCore to be created
java.util.concurrent.ExecutionException:
org.apache.solr.common.SolrException: Unable to create core
[bb-catalog-material_shard1_replica1]
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at org.apache.solr.core.CoreContainer.lambda$load$14(CoreContainer.java:671)
at
com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.solr.common.SolrException: Unable to create core
[bb-catalog-material_shard1_replica1]
at
org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1053)
at org.apache.solr.core.CoreContainer.lambda$load$13(CoreContainer.java:642)
at
com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:197)
... 5 more
Caused by: org.apache.solr.common.SolrException: Error opening new searcher
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1008)
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:863)
at
org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1037)
... 7 more
Caused by: org.apache.solr.common.SolrException: Error opening new searcher
at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:2095)
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:2215)
at org.apache.solr.core.SolrCore.initSearcher(SolrCore.java:1091)
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:980)
... 9 more
Caused by: java.lang.IllegalArgumentException: cannot change DocValues type
from SORTED to SORTED_SET for field "search_vendor_sku"
at org.apache.lucene.index.FieldInfo.setDocValuesType(FieldInfo.java:212)
at
org.apache.lucene.index.FieldInfos$Builder.addOrUpdateInternal(FieldInfos.java:438)
at org.apache.lucene.index.FieldInfos$Builder.add(FieldInfos.java:451)
at org.apache.lucene.index.FieldInfos$Builder.add(FieldInfos.java:446)
at org.apache.lucene.index.FieldInfos$Builder.add(FieldInfos.java:380)
at
org.apache.lucene.index.MultiFields.getMergedFieldInfos(MultiFields.java:268)
at
org.apache.solr.index.SlowCompositeReaderWrapper.getFieldInfos(SlowCompositeReaderWrapper.java:263)
at
org.apache.solr.search.SolrIndexSearcher.<init>(SolrIndexSearcher.java:279)
at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:2063)

We have not modified the field definition for search_vendor_sku

<field name="display_vendor_sku" type="string" indexed="false"
stored="true" required="false" multiValued="false"
docValues="true"/><field name="search_vendor_sku"
type="text_lowercase" indexed="true" stored="true" required="false"
multiValued="false" docValues="false"/>




Both collections had the same error

A third collection with the same configuration restarted with no problems.

My theory is that the field was changed from a string to text_lowercase,
and docValues set to false then data was reloaded. No issues on the reload,
but maybe there were some docValues data left over from its previous
incarnation as a string?

This is dev, I just want to understand what happened so we can prevent it
in production

-- 


This message and any attachment are confidential and may be privileged or 
otherwise protected from disclosure. If you are not the intended recipient, 
you must not copy this message or attachment or disclose the contents to 
any other person. If you have received this transmission in error, please 
notify the sender immediately and delete the message and any attachment 
from your system. Merck KGaA, Darmstadt, Germany and any of its 
subsidiaries do not accept liability for any omissions or errors in this 
message which may arise as a result of E-Mail-transmission or for damages 
resulting from any unauthorized changes of the content of this message and 
any attachment thereto. Merck KGaA, Darmstadt, Germany and any of its 
subsidiaries do not guarantee that this message is free of viruses and does 
not accept liability for any damages caused by any virus transmitted 
therewith.

Click http://www.emdgroup.com/disclaimer to access the German, French, 
Spanish and Portuguese versions of this disclaimer.

Re: Collections Fail to load after Solr Restart

[Erick Erickson <er...@gmail.com>]

bq: My theory is that the field was changed from a string to
text_lowercase, and docValues set to false then data was reloaded

That would do it. For most changes in the schema, you should expect to
_have_ to completely blow away the index and re-index from scratch.
The underlying structures can be totally different and even absent
that, the changes often mean that searchers don't work any more. To
prevent downtime, people will index to a brand new collection and then
atomically switch by creating an alias when they need to change the
schema.

Text-based fields are ineligible for docValues as well so indeed the
docValues field will be set to "false", but this may change in future,
see: https://issues.apache.org/jira/browse/SOLR-8362

The error you're reporting is hard to understand, there's been a JIRA
set up for that here: https://issues.apache.org/jira/browse/SOLR-12002

There are very, very few situations in which you want to change the
fieldType of a field without re-indexing. In your case, pre-existing
documents with any mixed case would _not_ be searchable after the
change, i.e. the token in the index for a pre-existing document could
be something like the following _single tokens_: "WeIrd", or even
"Weird Science" (string types do not break up the input for instance).
So after the change above (assuming text_lowercase is something like
WhitespaceTokenizer followed by LowercaseFilter) you could never find
either of these as the _query_ tokens would be
"weird" and "weird" "science" (this latter as two tokens with the
default operator between).

People sometimes use KeywordTokenizer followed by LowercaseFilter to
preserve the single-token nature of inputs FWIW.

Finally, you start getting errors like this when any pre-existing
segments are around after the change. Therefore I recommend either
creating a new collection or removing all the data directories when
people make changes to the schema..

That all applies to changing _existing_ fields. If you are simply
adding or removing fields then you can change the schema without
re-indexing if your app can tolerate some docs with and some docs
without a field.

Best,
Erick



On Fri, Feb 16, 2018 at 8:21 AM, Webster Homer <we...@sial.com> wrote:
> Yesterday I restarted a development solrcloud. After the cloud restarted 2
> collections failed to come back.
>
> I see this in the log:
> 2018-02-16 15:31:16.684 ERROR
> (coreLoadExecutor-6-thread-1-processing-n:ae1c-ecomdev-msc02:8983_solr) [
>  ] o.a.s.c.CachingDirectoryFactory Error closing
> directory:org.apache.solr.common.SolrException: Timeout waiting for all
> directory ref counts to be released - gave up waiting on
> CachedDir<<refCount=1;path=/var/solr/data/bb-catalog-material_shard1_replica1/data/index;done=false>>
> at
> org.apache.solr.core.CachingDirectoryFactory.close(CachingDirectoryFactory.java:178)
> at org.apache.solr.core.SolrCore.close(SolrCore.java:1633)
> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1000)
> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:863)
> at
> org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1037)
> at org.apache.solr.core.CoreContainer.lambda$load$13(CoreContainer.java:642)
> at
> com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:197)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
>
>
> right after this I also see the following error:
> 2018-02-16 15:31:16.713 ERROR
> (coreContainerWorkExecutor-2-thread-1-processing-n:ae1c-ecomdev-msc02:8983_solr)
> [   ] o.a.s.c.CoreContainer Error waiting for SolrCore to be created
> java.util.concurrent.ExecutionException:
> org.apache.solr.common.SolrException: Unable to create core
> [bb-catalog-material_shard1_replica1]
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at org.apache.solr.core.CoreContainer.lambda$load$14(CoreContainer.java:671)
> at
> com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.solr.common.SolrException: Unable to create core
> [bb-catalog-material_shard1_replica1]
> at
> org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1053)
> at org.apache.solr.core.CoreContainer.lambda$load$13(CoreContainer.java:642)
> at
> com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:197)
> ... 5 more
> Caused by: org.apache.solr.common.SolrException: Error opening new searcher
> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1008)
> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:863)
> at
> org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1037)
> ... 7 more
> Caused by: org.apache.solr.common.SolrException: Error opening new searcher
> at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:2095)
> at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:2215)
> at org.apache.solr.core.SolrCore.initSearcher(SolrCore.java:1091)
> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:980)
> ... 9 more
> Caused by: java.lang.IllegalArgumentException: cannot change DocValues type
> from SORTED to SORTED_SET for field "search_vendor_sku"
> at org.apache.lucene.index.FieldInfo.setDocValuesType(FieldInfo.java:212)
> at
> org.apache.lucene.index.FieldInfos$Builder.addOrUpdateInternal(FieldInfos.java:438)
> at org.apache.lucene.index.FieldInfos$Builder.add(FieldInfos.java:451)
> at org.apache.lucene.index.FieldInfos$Builder.add(FieldInfos.java:446)
> at org.apache.lucene.index.FieldInfos$Builder.add(FieldInfos.java:380)
> at
> org.apache.lucene.index.MultiFields.getMergedFieldInfos(MultiFields.java:268)
> at
> org.apache.solr.index.SlowCompositeReaderWrapper.getFieldInfos(SlowCompositeReaderWrapper.java:263)
> at
> org.apache.solr.search.SolrIndexSearcher.<init>(SolrIndexSearcher.java:279)
> at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:2063)
>
> We have not modified the field definition for search_vendor_sku
>
> <field name="display_vendor_sku" type="string" indexed="false"
> stored="true" required="false" multiValued="false"
> docValues="true"/><field name="search_vendor_sku"
> type="text_lowercase" indexed="true" stored="true" required="false"
> multiValued="false" docValues="false"/>
>
>
>
>
> Both collections had the same error
>
> A third collection with the same configuration restarted with no problems.
>
> My theory is that the field was changed from a string to text_lowercase,
> and docValues set to false then data was reloaded. No issues on the reload,
> but maybe there were some docValues data left over from its previous
> incarnation as a string?
>
> This is dev, I just want to understand what happened so we can prevent it
> in production
>
> --
>
>
> This message and any attachment are confidential and may be privileged or
> otherwise protected from disclosure. If you are not the intended recipient,
> you must not copy this message or attachment or disclose the contents to
> any other person. If you have received this transmission in error, please
> notify the sender immediately and delete the message and any attachment
> from your system. Merck KGaA, Darmstadt, Germany and any of its
> subsidiaries do not accept liability for any omissions or errors in this
> message which may arise as a result of E-Mail-transmission or for damages
> resulting from any unauthorized changes of the content of this message and
> any attachment thereto. Merck KGaA, Darmstadt, Germany and any of its
> subsidiaries do not guarantee that this message is free of viruses and does
> not accept liability for any damages caused by any virus transmitted
> therewith.
>
> Click http://www.emdgroup.com/disclaimer to access the German, French,
> Spanish and Portuguese versions of this disclaimer.