You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Varun Thacker (JIRA)" <ji...@apache.org> on 2014/12/13 09:54:13 UTC
[jira] [Commented] (SOLR-6845) figure out why suggester causes slow
startup - even when not used
[ https://issues.apache.org/jira/browse/SOLR-6845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14245257#comment-14245257 ]
Varun Thacker commented on SOLR-6845:
-------------------------------------
Excerpt from the logs -
{code}
16160 [coreLoadExecutor-5-thread-2] INFO org.apache.solr.handler.component.SpellCheckComponent – Initializing spell checkers
16161 [coreLoadExecutor-5-thread-2] INFO org.apache.solr.spelling.DirectSolrSpellChecker – init: {name=default,field=text,classname=solr.DirectSolrSpellChecker,distanceMeasure=internal,accuracy=0.5,maxEdits=2,minPrefix=1,maxInspections=5,minQueryLength=4,maxQueryFrequency=0.01}
16162 [coreLoadExecutor-5-thread-2] INFO org.apache.solr.handler.component.SpellCheckComponent – No queryConverter defined, using default converter
16164 [coreLoadExecutor-5-thread-2] INFO org.apache.solr.handler.component.SuggestComponent – Initializing SuggestComponent
16164 [coreLoadExecutor-5-thread-2] INFO org.apache.solr.spelling.suggest.SolrSuggester – init: {name=mySuggester,lookupImpl=FuzzyLookupFactory,dictionaryImpl=DocumentDictionaryFactory,field=cat,weightField=price,suggestAnalyzerFieldType=string}
16164 [coreLoadExecutor-5-thread-2] INFO org.apache.solr.spelling.suggest.SolrSuggester – Dictionary loaded with params: {name=mySuggester,lookupImpl=FuzzyLookupFactory,dictionaryImpl=DocumentDictionaryFactory,field=cat,weightField=price,suggestAnalyzerFieldType=string}
16164 [coreLoadExecutor-5-thread-2] INFO org.apache.solr.common.cloud.ZkStateReader – Load collection config from:/collections/solr6606
16166 [coreLoadExecutor-5-thread-2] INFO org.apache.solr.common.cloud.ZkStateReader – path=/collections/solr6606 configName=solr6606 specified config exists in ZooKeeper
16186 [coreLoadExecutor-5-thread-2] INFO org.apache.solr.handler.ReplicationHandler – Commits will be reserved for 10000
16187 [searcherExecutor-6-thread-1] INFO org.apache.solr.core.SolrCore – QuerySenderListener sending requests to Searcher@6b1b4cac[solr6606_shard1_replica1] main{StandardDirectoryReader(segments_5q:2567:nrt _9g(4.10.2):C794000 _io(4.10.2):C760000 _sf(4.10.2):C796000 _yk(4.10.2):C543000 _ze(4.10.2):C72000 _zf(4.10.2):C1000 _zg(4.10.2):C2000 _zh(4.10.2):C3000 _zi(4.10.2):C3000 _zj(4.10.2):C2121 _zk(4.10.2):C2879 _zl(4.10.2):C3000 _zm(4.10.2):C2000)}
16226 [searcherExecutor-6-thread-1] INFO org.apache.solr.core.SolrCore – [solr6606_shard1_replica1] webapp=null path=null params={event=firstSearcher&q=static+firstSearcher+warming+in+solrconfig.xml&distrib=false} hits=0 status=0 QTime=39
16227 [searcherExecutor-7-thread-1] INFO org.apache.solr.core.SolrCore – [collection1] webapp=null path=null params={event=firstSearcher&q=static+firstSearcher+warming+in+solrconfig.xml&distrib=false} hits=0 status=0 QTime=66
16227 [searcherExecutor-6-thread-1] INFO org.apache.solr.core.SolrCore – QuerySenderListener done.
16227 [searcherExecutor-7-thread-1] INFO org.apache.solr.core.SolrCore – QuerySenderListener done.
16227 [searcherExecutor-6-thread-1] INFO org.apache.solr.handler.component.SpellCheckComponent – Loading spell index for spellchecker: default
16227 [searcherExecutor-7-thread-1] INFO org.apache.solr.handler.component.SpellCheckComponent – Loading spell index for spellchecker: default
16228 [searcherExecutor-6-thread-1] INFO org.apache.solr.handler.component.SpellCheckComponent – Loading spell index for spellchecker: wordbreak
16228 [searcherExecutor-7-thread-1] INFO org.apache.solr.handler.component.SpellCheckComponent – Loading spell index for spellchecker: wordbreak
16228 [searcherExecutor-6-thread-1] INFO org.apache.solr.handler.component.SuggestComponent – Loading suggester index for: mySuggester
16228 [searcherExecutor-7-thread-1] INFO org.apache.solr.handler.component.SuggestComponent – Loading suggester index for: mySuggester
16229 [searcherExecutor-7-thread-1] INFO org.apache.solr.spelling.suggest.SolrSuggester – reload()
16228 [searcherExecutor-6-thread-1] INFO org.apache.solr.spelling.suggest.SolrSuggester – reload()
16229 [searcherExecutor-6-thread-1] INFO org.apache.solr.spelling.suggest.SolrSuggester – build()
16229 [searcherExecutor-7-thread-1] INFO org.apache.solr.spelling.suggest.SolrSuggester – build()
16249 [searcherExecutor-7-thread-1] INFO org.apache.solr.core.SolrCore – [collection1] Registered new searcher Searcher@2c8633e1[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
{code}
The SuggestComponent init's correctly. It's only when the firstSearcher event gets fired that the spellchecker and the suggester gets built. This is where the problem lies.
We should just comment out the {{<str name="q">static firstSearcher warming in solrconfig.xml</str>}} query from the {{firstSearcher}} query list, just as how the queries are commented out from the {{newSearcher}} event listener.
We should also add a comment here saying that adding queries here does the following internally and would lead to longer load times.
1. Builds the suggester, spellchecker and any other components which register itself to new searcher events.
2. When an entry contains a sort/facet/function query and the field is not an docValues field then it UnInverts the fields and puts it into the FieldCache
3. Adds entries to the other Solr caches.
The multiple reload and build commands is because I have 2 cores on my machine.
> figure out why suggester causes slow startup - even when not used
> -----------------------------------------------------------------
>
> Key: SOLR-6845
> URL: https://issues.apache.org/jira/browse/SOLR-6845
> Project: Solr
> Issue Type: Bug
> Reporter: Hoss Man
>
> SOLR-6679 was filed to track the investigation into the following problem...
> {panel}
> The stock solrconfig provides a bad experience with a large index... start up Solr and it will spin at 100% CPU for minutes, unresponsive, while it apparently builds a suggester index.
> ...
> This is what I did:
> 1) indexed 10M very small docs (only takes a few minutes).
> 2) shut down Solr
> 3) start up Solr and watch it be unresponsive for over 4 minutes!
> I didn't even use any of the fields specified in the suggester config and I never called the suggest request handler.
> {panel}
> ..but ultimately focused on removing/disabling the suggester from the sample configs.
> Opening this new issue to focus on actually trying to identify the root problem & fix it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org