8.5.1 LogReplayer extremely slow

[Markus Jelsma <ma...@openindex.io>]

Hello,

Our main Solr text search collection broke down last night (search was still working fine), every indexing action timed out with the Solr master spending most of its time in Java regex. One shard has only one replica left for queries and it stays like that. I have copied both shard's leader to local to see what is going on.

One shard is fine but the other has a replica with has about 600MB of data to replay and it is extremely slow. Using the VisualVM sampler i find that the replayer is also spending almost all time in dealing with Java regex (stack trace below). Is this to be expected? And what is it actually doing? Where do the TokenFilters come from?

I had a old but clean collection on the same cluster and started indexing to it to see what is going on but it too timed out due to Java regex. This is weird, because locally i have no problem indexing a million records in a 8.5.1 collection, and the broken down cluster has been running fine for over a month.

A note, this index uses PreAnalyzedField, so i would expect no analysis or whatsoever, certainly no regex.

Thanks,
Markus

"replayUpdatesExecutor-3-thread-1-processing-n:127.0.1.1:8983_solr x:sitesearch_shard2_replica_t2 c:sitesearch s:shard2 r:core_node4" #222 prio=5 os_prio=0 cpu=239207,44ms elapsed=239,50s tid=0x00007ffde0057000 nid=0x24f5 runnable  [0x00007ffeedd0f000]
   java.lang.Thread.State: RUNNABLE
        at java.util.regex.Pattern$GroupTail.match(java.base@11.0.7/Pattern.java:4863)
        at java.util.regex.Pattern$CharPropertyGreedy.match(java.base@11.0.7/Pattern.java:4306)
        at java.util.regex.Pattern$GroupHead.match(java.base@11.0.7/Pattern.java:4804)
        at java.util.regex.Pattern$CharPropertyGreedy.match(java.base@11.0.7/Pattern.java:4306)
        at java.util.regex.Pattern$Start.match(java.base@11.0.7/Pattern.java:3619)
        at java.util.regex.Matcher.search(java.base@11.0.7/Matcher.java:1729)
        at java.util.regex.Matcher.find(java.base@11.0.7/Matcher.java:746)
        at org.apache.lucene.analysis.pattern.PatternReplaceFilter.incrementToken(PatternReplaceFilter.java:71)
        at org.apache.lucene.analysis.miscellaneous.TrimFilter.incrementToken(TrimFilter.java:42)
        at org.apache.lucene.analysis.FilteringTokenFilter.incrementToken(FilteringTokenFilter.java:49)
        at org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:812)
        at org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:442)
        at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:406)
        at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:250)
        at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:495)
        at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1594)
        at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1586)
        at org.apache.solr.update.DirectUpdateHandler2.updateDocOrDocValues(DirectUpdateHandler2.java:979)
        at org.apache.solr.update.DirectUpdateHandler2.doNormalUpdate(DirectUpdateHandler2.java:345)
        at org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:292)
        at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:239)
        at org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:76)
        at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55)
        at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:259)
        at org.apache.solr.update.processor.DistributedUpdateProcessor.doVersionAdd(DistributedUpdateProcessor.java:489)
        at org.apache.solr.update.processor.DistributedUpdateProcessor.lambda$versionAdd$0(DistributedUpdateProcessor.java:339)
        at org.apache.solr.update.processor.DistributedUpdateProcessor$$Lambda$631/0x0000000840670c40.apply(Unknown Source)
        at org.apache.solr.update.VersionBucket.runWithLock(VersionBucket.java:50)
        - locked <0x00000000a7df5620> (a org.apache.solr.update.VersionBucket)
        at org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:339)
        at org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:225)
        at org.apache.solr.update.processor.DistributedZkUpdateProcessor.processAdd(DistributedZkUpdateProcessor.java:245)
        at org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:103)
        at org.apache.solr.update.UpdateLog$LogReplayer.lambda$execute$1(UpdateLog.java:2025)
        at org.apache.solr.update.UpdateLog$LogReplayer$$Lambda$629/0x0000000840672c40.run(Unknown Source)