You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@solr.apache.org by "SpookyJumpyBeans (via GitHub)" <gi...@apache.org> on 2026/10/16 03:42:53 UTC
[PR] SOLR-18548: Overseer collection lock leaks when marking an async task as running fails [solr]
SpookyJumpyBeans opened a new pull request, #5167:
URL: https://github.com/apache/solr/pull/5167
https://issues.apache.org/jira/browse/SOLR-18548
# Description
`OverseerTaskProcessor` takes a task's lock before storing its async ID in the running map. If that `runningMap.put` fails with a `KeeperException` other than `NodeExistsException` (for example an `OperationTimeoutException`), the exception reaches the outer loop with the lock still held. No Runner exists yet to release it, so every later task needing the same lock stays blocked until the Overseer restarts.
This has the same symptom as SOLR-17754, but a different path: that fix addressed a race with the runner pool, while this one happens before a Runner is created.
# Solution
Release the lock before rethrowing, so the outer loop keeps its existing handling (returning on session expiry, otherwise logging and backing off for a second before the next attempt):
```java
} catch (KeeperException e) {
// No Runner was created, so nothing else will release the lock.
lock.unlock();
throw e;
}
```
# Notes
- `RuntimeException` isn't caught here on purpose. The outer loop retries those immediately without backing off, so releasing the lock there could turn a persistent error into a busy loop, where today the task just stays blocked.
- If a timed-out `runningMap.put` actually reached ZooKeeper, the retry takes the existing `NodeExistsException` branch, which already releases the lock. That branch's handling of the leftover async ID is unchanged by this PR.
# Tests
Added `testRunningMapFailureReleasesCollectionLock` to `OverseerCollectionConfigSetProcessorTest`. It makes `runningMap.put` throw `OperationTimeoutException` for an async task, stops the processor, and checks that the collection lock can be acquired again through the real `LockTree`. It fails without the fix and passes with it. The real-handler setup it shares with `testLockTaskExceptionFailsAsyncTask` is moved into two small helpers.
Also ran the full `OverseerCollectionConfigSetProcessorTest` and `MultiThreadedOCPTest` suites, and `./gradlew :solr:core:check -x test`.
# Checklist
- [x] I have reviewed the guidelines for [How to Contribute](https://github.com/apache/solr/blob/main/CONTRIBUTING.md) and my code conforms to the standards described there to the best of my ability.
- [x] I have created a Jira issue and added the issue ID to my pull request title.
- [x] I have given Solr maintainers [access](https://help.github.com/en/articles/allowing-changes-to-a-pull-request-branch-created-from-a-fork) to contribute to my PR branch. (optional but recommended, not available for branches on forks living under an organisation)
- [x] I have developed this patch against the `main` branch.
- [ ] I have run `./gradlew check`.
- [x] I have added tests for my changes.
- [ ] I have added documentation for the [Reference Guide](https://github.com/apache/solr/tree/main/solr/solr-ref-guide)
- [x] I have added a [changelog entry](https://github.com/apache/solr/blob/main/dev-docs/changelog.adoc) for my change
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-18548: Overseer collection lock leaks when marking an async task as running fails [solr]
Posted by "nick-boss-tech (via GitHub)" <gi...@apache.org>.
nick-boss-tech commented on PR #5167:
URL: https://github.com/apache/solr/pull/5167#issuecomment-6390058573
**The lock-release fix looks sound; one P3 rationale/test issue found.**
The cleanup comment says an existing running marker would make retries fail with NodeExistsException. However, DistributedMap.put uses overwrite-on-exists behavior, so a successful retry can proceed without deleting that marker.
The test proves lock availability and checks that remove was called, but does not establish the claimed retry failure. Correct the rationale and, if retaining cleanup as retry recovery, test a write that succeeds before timing out followed by successful task execution.
AI-generated static review.
Thanks for submitting this!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-18548: Overseer collection lock leaks when marking an async task as running fails [solr]
Posted by "SpookyJumpyBeans (via GitHub)" <gi...@apache.org>.
SpookyJumpyBeans commented on PR #5167:
URL: https://github.com/apache/solr/pull/5167#issuecomment-6393816419
Thanks, you're right: `DistributedMap.put` overwrites an existing node (`makePath` with `failOnExists=false`), so removing the async ID was unnecessary. I reverted that commit and corrected the description.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org