You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by shushuai zhu <ss...@yahoo.com.INVALID> on 2020/11/23 04:47:08 UTC
Hitting solr throttling for ingestion
Hi Experts,
We are using solr 8.4 (none cloud). When ingesting data with multiple processes to one core in a solr node, we are hitting some throttling: the max ingestion rate achieved is about 47K docs per second with 17 posting processes; each doc is about 250 bytes; the CPU utilization rate is only 20% and I/O about 6%. When increasing the posting processes, the posting will start failing. With solr 6.6, such issue does not happen: increasing posting processes will increase CPU/IO utilization rates to be close to 100% then start failing.
Below are some relevant configurations specified in solrconfig.xml:
<indexConfig> <maxIndexingThreads>16</maxIndexingThreads> <ramBufferSizeMB>1024</ramBufferSizeMB> <mergePolicyFactory class="org.apache.solr.index.TieredMergePolicyFactory"> <int name="maxMergeAtOnce">10</int> <int name="segmentsPerTier">10</int> <double name="maxMergedSegmentMB">4096</double> <double name="noCFSRatio">0.1</double> <double name="maxCFSSegmentSizeMB">4096</double> </mergePolicyFactory> <lockType>${solr.lock.type:native}</lockType> <infoStream>true</infoStream></indexConfig>
<updateHandler class="solr.DirectUpdateHandler2"> <updateLog> <str name="dir">${solr.ulog.dir:}</str> <int name="numVersionBuckets">${solr.ulog.numVersionBuckets:65536}</int> </updateLog>
<autoCommit> <maxTime>${solr.autoCommit.maxTime:120000}</maxTime> <openSearcher>false</openSearcher> </autoCommit>
<autoSoftCommit> <maxTime>${solr.autoSoftCommit.maxTime:5000}</maxTime> </autoSoftCommit></updateHandler>
It seems maxIndexingThreads is no longer supported in solr 8? Any idea to break the solr throttling? Thanks.
Shushuai