Solr 8.1.1 Indexing issue while migrating Java8 -> Java11

["anup.junagade" <an...@searshc.com>]

 
We are trying to migrate our SOLR 8.1.1 cluster from OpenJDK Java 8 to
OpenJDK Java 11 and are facing issues with Indexing. While our indexing is
happening flawlessly on Java 8, it crawls or maybe I should say it stalls
with Java 11.
Any pointers/help is appreciated.
 
*Symptoms*
 
With OpenJDK 11 and SOLR 8.1.1 we see that for the first 30 minutes response
times for updates similar to our current implementation (OpenJDK 8 and SOLR
8.1.1). It has to be noted that there are no read queries being executed at
the time of indexing
On the OpenJDK 11 implementation, the qtp active threads continuously
increasing to thousands while OpenJDK 8 implementation stops after
approximately going up to 150.
On the On the OpenJDK 11 implementation, the classes loaded start at a very
high number and stay there as opposed to  OpenJDK 8  implementation where
the number of classes loaded are small to begin with and remains under
control. I believe the qtp threads in wait state mentioned above are causing
this symptom
Attachment 1:  OpenJDK 8 vs OpenJDK 8 key metrics
<https://drive.google.com/open?id=1YRlP-vBrxAJ7NpyY_DkjPemU6gigsYkt>  
Attachment 2:   OpenJDK 8 vs OpenJDK 8 waiting QTP Threads
<https://drive.google.com/open?id=10eYjlveQyuepTbwU6ZByIgGX4dRNCkDe>  
Attachment 3:  OpenJDK 11 Thread dump
<https://drive.google.com/open?id=1aLaWDR9vPMSqPavO1z8ZSzuLocUkp7L3>  
 
 
*Following are the key configuration of our application.*
 
Index Size: 8 GB/shard
Total no of Documents in Solr cluster: 70 Million
Average Document size: 15 KB
JSON Payload for each update contains: 50 docs
Average Time Taken to post for 50 Docs: 300 milli seconds
Average Rate at which documents are being posted to SOLR: 7500 requests per
second
No of shards in the Cluster: 10 (No Replicas)
CPUs: 32
Memory: 128 GB
Heap allocated: 32 GB
SOLR Client: 8.1.1
ZK Ensemble: 3
 
<mergePolicyFactory class="org.apache.solr.index.TieredMergePolicyFactory">
          <int name="maxMergeAtOnce">100</int>
          <int name="segmentsPerTier">150</int>
</mergePolicyFactory>
 
<autoSoftCommit>
            <maxTime>480000</maxTime>
</autoSoftCommit>
<autoCommit>
            <maxTime>180000</maxTime>
            <openSearcher>false</openSearcher>
</autoCommit>
 
GC and ZK Settings
 
-DzkClientTimeout=300000
-DzkHost= XXXX,YYYY,ZZZZ
-XX:+PrintGCDetails
-XX:+UseG1GC
-XX:+UseStringDeduplication
-XX:ConcGCThreads=8
-XX:InitiatingHeapOccupancyPercent=70
-XX:MaxGCPauseMillis=200
-XX:ParallelGCThreads=32
-XX:PermSize=512m
-Xlog:gc*:file=/var/solr/logs/solr_gc.log:time,
uptime:filecount=9,
filesize=20M
-Xms32g-Xmx32g
-Xss256k-Xss256k
-verbose:gc
 

 



--
Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html