[jira] [Comment Edited] (SOLR-18548) Overseer collection lock leaks when marking an async task as running fails

["Boyi Dai (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SOLR-18548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18128924#comment-18128924 ] 

Boyi Dai edited comment on SOLR-18548 at 10/19/26 11:01 PM:
------------------------------------------------------------

[~psalagnac] 

 Thanks for taking a look! I found it by analyzing the code rather than in a real cluster. I haven't seen it in production.
 
The failure needs runningMap.put to fail with something other than NodeExistsException, e.g. a ConnectionLossException that outlasts the ZK client's retries, or an OperationTimeoutException, during a ZK hiccup. That's rare, but when it happens the lock is never released, so it doesn't clear up on its own: every later task for that collection stays blocked until the Overseer restarts. That's why I thought it was worth fixing even without a field report.


was (Author: JIRAUSER314998):
[~psalagnac] 

 
Thanks for taking a look! I found it by analyzing the code rather than in a real cluster. I haven't seen it in production.
 
The failure needs runningMap.put to fail with something other than NodeExistsException, e.g. a ConnectionLossException that outlasts the ZK client's retries, or an OperationTimeoutException, during a ZK hiccup. That's rare, but when it happens the lock is never released, so it doesn't clear up on its own: every later task for that collection stays blocked until the Overseer restarts. That's why I thought it was worth fixing even without a field report.

> Overseer collection lock leaks when marking an async task as running fails
> --------------------------------------------------------------------------
>
>                 Key: SOLR-18548
>                 URL: https://issues.apache.org/jira/browse/SOLR-18548
>             Project: Solr
>          Issue Type: Bug
>          Components: SolrCloud
>    Affects Versions: 10.0
>            Reporter: Boyi Dai
>            Priority: Minor
>              Labels: pull-request-available
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> OverseerTaskProcessor takes the lock for a task (collection, shard or replica) before storing the task’s async ID in the running map in ZooKeeper. If that {{runningMap.put}} fails, only {{NodeExistsException}} and {{InterruptedException}} release the lock. Any other {{KeeperException}} (for example {{{}OperationTimeoutException{}}}, or a {{ConnectionLossException}} after retries) or a {{RuntimeException}} goes to the outer loop’s handler with the lock still held.
> No Runner has been created at that point, so nothing ever releases the lock. Every later task that needs the same lock is blocked until the Overseer restarts, so a single transient ZooKeeper error during that write can stall all collection API operations on that collection.
> This has the same symptom as SOLR-17754 (a lock left held so tasks never run), but a different cause: SOLR-17754 fixed a race with the runner thread pool, while this happens when the ZooKeeper write fails before a Runner exists.
> Expected: the lock is released whenever the task is not handed to a Runner, so the task can be retried and other tasks for the same collection can proceed. A fix with a test is coming in a PR.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org