Potential bug in task list management

[Reuben Thompson <re...@acresoftware.com>]

Hi, all,

Long time user, first time caller... I've been using Solr on and off since 2008.

We've identified a potential resource leak in the task management subsystem that we believe is the cause of crashes of long running nodes. Before raising a bug I thought I should check with this list whether it was known or I'm barking up the wrong tree.

Essentially, whenever a query task is abnormally ended, ie either the client times out and closes the connection, the query hits the timeAllowed or cpuAllowed limit, or the task is cancelled through the /solr/collection/tasks/cancel?queryUUID= mechanism, the task is never or almost never removed from the list of tasks returned by the /v2/collections/collection/tasks/list endpoint.

We also suspect that other resources are not always returned in these circumstances, even after hours or days as the heap continues to grow in a way far, far greater than would be expected from the size of the task list. This leads to an ever-increasing number of tasks in the list, meaning that iterating it takes longer and longer and eventual slowdown such that the number of tasks waiting grows to the extent that the node becomes unresponsive and restarts. We also see inconsistent lists of tasks on each node as this happens.

As this is taking approx 2 months to become a problem on our prod nodes as we don't fail many transactions, I wrote a repro script against one of our collections and ran it on a deliberately CPU constrained SolrCloud locally with three nodes, each with 4GB of heap available and approx 1GB of data to deliberately cause tasks to time out. This resulted in over 10,000 entries in the task list, none of which were active, after I stopped the script. Leaving the nodes running for a further 12 hours saw no reduction in the number of listed tasks.

I've tried this both against the 9.4 we run in production and 9.8.0 just to see if it's improved, and although 9.8 is noticeably faster (nice work), the same thing happens.

Any ideas?

Thanks in advance,



Reuben







Reuben Thompson

VP Product Innovation


e:  reuben.thompson@acresoftware.com<ma...@acresoftware.com>

w: acresoftware.com



[cid:822934a0-d587-4f8a-9b24-0fd01d9068a6]

Re: Potential bug in task list management

[Chris Hostetter <ho...@fucit.org>]

: Essentially, whenever a query task is abnormally ended, ie either the 
: client times out and closes the connection, the query hits the 
: timeAllowed or cpuAllowed limit, or the task is cancelled through the 
: /solr/collection/tasks/cancel?queryUUID= mechanism, the task is never or 
: almost never removed from the list of tasks returned by the 
: /v2/collections/collection/tasks/list endpoint.

I'm not really familiar with the "task" management API and the cancelng 
queries, not have I tried to reproduce the behavior you are describing, 
but based on a skim of the only test I see that involves canceling 
queries, i don't see anything in that test that would rule out what you're 
descrbing.

TestTaskManagement.testNonExistentQuery
 - just asserts a 404 when trying to cancel an non existent UUID

TestTaskManagement.testCancellationQuery
 - runs some queries in background threads and then cancels them
 - nothing about the queries ensures they are still around to be canceled
 - test only asserts that the number of queries it created equals the 
   number of successs + failures in trying to cancel
 - so even if the 100% of the queries never ran, and were never tracked, 
   this test would pass
 - and nothing in the test confirms that any tasks which *might* have been 
   tracked are removed from the list at the end of the test

TestTaskManagement.testListCancellableQueries
 - runs 50 queries in background threads and then lists current tasks
 - only asserts that the number of items in the is: 0 <= n <= 50 
 - so again: if 100% of the queries never make it to solr the test passes
 - if 100% of the queries are stuck in the list forever, the test passes


So yeah.  There's really not much that this test actaully proves.

Can you please file a Jira with the details of your observations?



If you're up for it, here's how i would approach fixing the test:


1) write a custom SearchComponent that checks for some "blockTest=true" 
request param, and if it's set...

  - calls release() on a "public static final Semaphore REQ_READY"
  - then calls acquire() on a "public static final Semaphore REQ_WAITS_FOR"

...before letting the request finish

2) register & use that component in a /blocking SearchHandler (either via 
a new configset, or via the APIs to add them at query time)


3) change the test logic:

  - use the new /blocking request handler and send blockTest=true on all N requests
  - wait to acquire(N) permits from REQ_READY before making any 
    assertions about the task list and/or canceling X requests
  - then and only then release(N-X) on REQ_WAITS_FOR
  - wait for all the background request responses, and assert that the 
canceld ones failed, and the other ones succeeded
  - then check the task list and confirm it's now empty.



-Hoss
http://www.lucidworks.com/