High thread count while writing collection

[DAVID MARTIN NIETO <dm...@viewnext.com>]

Hello,

We have a cluster of solr 8.2 and sometimes when we have a high load of incoming queries about solr and we have to reindex at the same time, the following happens:
- One of the nodes or several that make up the cluster of solr sees how its processes are triggered in the order of 1000 until reaching 5000 and even sometimes up to more than 10000 degrading the machine to the SO level with the consecuent loss of service, of that node.
- Looking at the detail of the thread dump we see that what appears in these cases are threads of this style:
qtp288994035-118554 (118554)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@471882ba

sun.misc.Unsafe.park (Native Method)
java.util.concurrent.locks.LockSupport.parkNanos (LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer $ ConditionObject.awaitNanos (AbstractQueuedSynchronizer.java:2078)
org.eclipse.jetty.util.BlockingArrayQueue.poll (BlockingArrayQueue.java:392)
org.eclipse.jetty.util.thread.QueuedThreadPool $ Runner.idleJobPoll (QueuedThreadPool.java:850)
org.eclipse.jetty.util.thread.QueuedThreadPool $ Runner.run (QueuedThreadPool.java:889)
java.lang.Thread.run (Thread.java:745)

Someone would know what could be due or how we could arrive at a specific cause. The solr or zookeeper logs do not show any problems at this time.

Example of one of this increasing:

[cid:5317ef95-a733-4218-b5d4-84b70324bc16]

The collection load batch that we use does not show any error either. Additionally, this upload of threads is not released after x minutes or hours but remains in time, descending in several days, which by thread timeouts I understand should disappear sooner.

Greetings and thank you for your help.


---------------------------------------------------------------------------------------------------------------
Este mensaje va dirigido únicamente a la(s) persona(s) y/o entidad(es) arriba relacionada(s). Puede contener información confidencial o legalmente protegida. Si no es usted el destinatario señalado, le rogamos borre del sistema inmediatamente el mensaje y sus copias. Asimismo le informamos que cualquier copia, divulgación, distribución o uso de los contenidos está prohibida.
---------------------------------------------------------------------------------------------------------------
This message is addressed only to the person (people) and / or entities listed above. It may contain confidential or legally protected information. If you are not the recipient indicated, please delete the message and its copies immediately from the system. We also inform that any copy, disclosure, distribution or use of the contents is forbidden
---------------------------------------------------------------------------------------------------------------
Viewnext, S.A. Domicilio Social: Avda. de Burgos 8-A 28036 de Madrid. telf: 913834060, Fax: 913834090. Reg. M. Madrid: Tomo 3238, Libro:0, Folio: 78, Seccion: 8ª, Hoja M-55112, N.I.F.: A-80157746