[jira] [Created] (SOLR-18348) `buildOnCommit=true` suggester rebuild blocks commit (and the client request) indefinitely with no timeout

["Eric Pugh (Jira)" <ji...@apache.org>]

Eric Pugh created SOLR-18348:
--------------------------------

             Summary: `buildOnCommit=true` suggester rebuild blocks commit (and the client request) indefinitely with no timeout
                 Key: SOLR-18348
                 URL: https://issues.apache.org/jira/browse/SOLR-18348
             Project: Solr
          Issue Type: Bug
          Components: Suggester
    Affects Versions: 9.10.1
            Reporter: Eric Pugh
            Assignee: Eric Pugh


h3. Problem

When a suggester is configured with `buildOnCommit=true`, a slow suggester rebuild does not just make that commit take longer in the background — it synchronously blocks the commit call itself, and therefore the client/update request waiting on it, with *no timeout*. Under sustained heavy indexing combined with throttled storage (e.g. an Azure disk after burst credits are exhausted), this manifests as outright request timeouts requiring manual operator intervention, rather than graceful degradation.

A customer hit this in production: ~4 million product documents being indexed against a suggester with an I/O-intensive `AnalyzingInfixSuggester` configuration, on a disk that had exhausted its burst-credit allowance. Indexing requests began timing out. Thread dumps showed threads stuck in:

{code}
"searcherExecutor-250-thread-3-processing-...-core_node4" ... RUNNABLE
  at org.apache.lucene.util.compress.LZ4.decompress(LZ4.java:138)
  at org.apache.lucene.codecs.lucene90.LZ4WithPresetDictCompressionMode$LZ4WithPresetDictDecompressor.decompress(...)
  at org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsReader...
  at org.apache.lucene.search.suggest.DocumentDictionary$DocumentInputIterator.next(DocumentDictionary.java:169)
  at org.apache.lucene.search.suggest.analyzing.AnalyzingInfixSuggester.build(AnalyzingInfixSuggester.java:346)
{code}

h3. Root cause

* \{{SuggestComponent.SuggesterListener.newSearcher()}} calls \{{suggester.build(core, newSearcher)}} synchronously — see [SuggestComponent.java|https://github.com/apache/solr/blob/releases/solr/9.10.1/solr/core/src/java/org/apache/solr/handler/component/SuggestComponent.java#L566-L571].
* This listener is invoked as a \{{newSearcherListener}} on \{{SolrCore}}'s \{{searcherExecutor}}, which is explicitly a *single-threaded* executor (see the comment "WARNING: this code assumes a single threaded executor" in \{{SolrCore.java}}). Warmup, all \{{newSearcherListener}}s, and searcher registration all run sequentially on this one thread.
* \{{DirectUpdateHandler2.commit()}} blocks the committing thread on \{{waitSearcher[0].get()}} — a no-arg, *no-timeout* \{{Future.get()}} — whenever \{{waitSearcher=true}}, which is the default for a normal \{{commit=true}} request (\{{CommitUpdateCommand.waitSearcher}} defaults to \{{true}}).
* Net effect: an arbitrarily slow \{{buildOnCommit}} rebuild (or any other slow \{{newSearcherListener}}) blocks the commit-issuing thread/HTTP request indefinitely, bounded only by client/proxy socket timeouts.

This exact hazard is already flagged in the reference guide ("Some lookup implementations may take a long time to build... using buildOnCommit or buildOnOptimize, particularly with a high frequency of soft commits is not recommended") but the current behavior is worse than "slow" — it can hang a commit indefinitely with no internal safeguard.

Note: \{{SpellCheckComponent}} has an analogous \{{buildOnCommit}}/\{{SpellCheckerListener}} pattern and likely shares this same architectural risk; filing this ticket scoped to \{{SuggestComponent}} only.

h3. Reproduction

Two JUnit tests demonstrate this deterministically using a test-only dictionary that simulates slow I/O (sleeping a fixed time per term, so total build time is bounded and reproducible):

* \{{SuggestComponentBuildOnCommitBlocksCommitTest}} — with \{{buildOnCommit=true}}, \{{commit()}} takes as long as the (artificially slow) suggester build.
* \{{SuggestComponentBuildOnCommitDisabledCommitStaysFastTest}} — same slow dictionary, \{{buildOnCommit=false}}: \{{commit()}} stays fast, isolating the cause to \{{buildOnCommit}} itself.

h3. Suggested direction (open to discussion)

* Run \{{buildOnCommit}}/\{{buildOnOptimize}} suggester rebuilds off the shared single-threaded \{{searcherExecutor}} (e.g. a dedicated executor) so they don't block warmup/registration/commit for other listeners.
* And/or decouple the rebuild from the commit's \{{waitSearcher}} path entirely, so a slow rebuild degrades to "suggestions are stale a bit longer" rather than "commit hangs."
* And/or add an explicit, documented timeout/async option for \{{buildOnCommit}}.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org