You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@solr.apache.org by "sarkaramrit1993 (via GitHub)" <gi...@apache.org> on 2026/10/13 20:18:52 UTC
[PR] SOLR-18525: Fix core create hang when its replica is removed during registration [solr]
sarkaramrit1993 opened a new pull request, #5124:
URL: https://github.com/apache/solr/pull/5124
https://issues.apache.org/jira/browse/SOLR-18525
Creating a core can hang forever if its replica is removed from state.json while `ZkController.register` is running. `register` installs `UnloadCoreOnDeletedWatcher`, and `registerDocCollectionWatcher` runs it inline on the creating thread. The watcher sees the replica is gone and calls `CoreContainer.unload`, which since SOLR-17700 waits on the pending core op for that name. That op is held by the same `CoreContainer.create` call, so it never finishes. Other notification threads then pile up on the synchronized `onStateChanged`.
Changes:
- The watcher now queues the unload with `CoreContainer.runAsync` instead of calling it directly. The unload waits for the create to release its op, then removes the core.
- It only queues once per watcher, and skips the unload if the core is already gone or now belongs to a different coreNodeName.
- At shutdown, a rejected task is logged and dropped.
Not changed:
- The coreNodeName check happens before `unload` takes the op, so a core unloaded and recreated under the same name in that window could still be hit. The old code had the same gap. Closing it needs a new CoreContainer method, so I left it out.
- If an unload is queued behind a slow create during shutdown, the async executor can wait up to 60s before interrupting it. Bounded, no hang.
Tests: new `CoreCreateReplicaRemovedDeadlockTest`, two cases (collection already watched on the node, and first replica on the node, where the inline call races the async notification). It pauses register with a custom UpdateLog in `recoverFromLog`, since `testing_beforeRegisterInZk` fires before register checks cluster state. Both cases hang without the fix and pass with it. Also ran DeleteReplicaTest, DeleteShardTest, MoveReplicaTest, ReplaceNodeTest, DeleteNodeTest, and `:solr:core:check -x test`.
@HoustonPutman this touches the unload path from SOLR-17700, would appreciate a look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-18525: Fix core create hang when its replica is removed during registration [solr]
Posted by "sarkaramrit1993 (via GitHub)" <gi...@apache.org>.
sarkaramrit1993 commented on PR #5124:
URL: https://github.com/apache/solr/pull/5124#issuecomment-6374420618
@HoustonPutman friendly ping when you get a chance. The hang comes from the core operation lock that [SOLR-17700](https://issues.apache.org/jira/browse/SOLR-17700) ([#3260](https://github.com/apache/solr/pull/3260)) added to unload, so you're the best person to check that this fix keeps what that change was protecting. Happy to rework it if you'd prefer a different approach.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-18525: Fix core create hang when its replica is removed during registration [solr]
Posted by "sarkaramrit1993 (via GitHub)" <gi...@apache.org>.
sarkaramrit1993 commented on PR #5124:
URL: https://github.com/apache/solr/pull/5124#issuecomment-6155014405
@epugh could you approve the CI run here when you get a chance? `gradlew check -x test` passes locally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-18525: Fix core create hang when its replica is removed during registration [solr]
Posted by "sarkaramrit1993 (via GitHub)" <gi...@apache.org>.
sarkaramrit1993 commented on PR #5124:
URL: https://github.com/apache/solr/pull/5124#issuecomment-6351390681
@janhoy since you looked at [SOLR-18525](https://issues.apache.org/jira/browse/SOLR-18525), would you mind approving the CI run and taking a look when you have a moment? cc @dsmiley since this touches the `ZkController` unload path. The test is now repeatable and asserts on behavior instead of timing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-18525: Fix core create hang when its replica is removed during registration [solr]
Posted by "sarkaramrit1993 (via GitHub)" <gi...@apache.org>.
sarkaramrit1993 commented on PR #5124:
URL: https://github.com/apache/solr/pull/5124#issuecomment-6400821568
Good catch. The identity check runs before `unload()` waits for the core op, so if the core is removed and a new one with the same name gets the op first, the queued unload acts on the replacement. The window is narrow, since it needs core-name reuse right at the handoff, but the effect is deleting a live replica's files, so it's worth closing.
Doing the check under the same op means a small `CoreContainer` change, e.g. an unload that only proceeds if the `coreNodeName` still matches. @HoustonPutman would you rather have that in this PR or as a follow-up?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-18525: Fix core create hang when its replica is removed during registration [solr]
Posted by "nick-boss-tech (via GitHub)" <gi...@apache.org>.
nick-boss-tech commented on PR #5124:
URL: https://github.com/apache/solr/pull/5124#issuecomment-6390984537
**One P2 finding:**
The new replica-identity check happens before unload() acquires the core-operation lock. While the queued task waits, cleanup can remove the old core and another CREATE can reuse its name. The delayed unload then acts on the replacement, including deleting its files. See [ZkController.java:2982–2986](https://github.com/apache/solr/blob/09cadc87d60a427902cccec4f1ffbca6e1852c88/solr/core/src/java/org/apache/solr/cloud/ZkController.java#L2982-L2986).
Keep the asynchronous fix, but check identity under the same lock as the unload, with a controlled replacement-race test.
AI-generated, static review.
Happy to produce a diff you can use or to open a follow-up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org