You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@solr.apache.org by "oyjhl (via GitHub)" <gi...@apache.org> on 2026/03/29 01:28:51 UTC
[I] SolrBackup may never finish if REQUESTSTATUS returns notfound for an accepted backup [solr-operator]
oyjhl opened a new issue, #824:
URL: https://github.com/apache/solr-operator/issues/824
# What happened
A `SolrBackup` can remain stuck in `InProgress` forever if:
1. the backup request is accepted ([operator backup submit](https://github.com/apache/solr-operator/blob/ca9d3c5c37a59f29570a6b49a8da5dc614aba75e/controllers/util/backup_util.go#L94-L109), [stable backup async ID](https://github.com/apache/solr-operator/blob/ca9d3c5c37a59f29570a6b49a8da5dc614aba75e/controllers/util/backup_util.go#L56-L58)),
2. the operator is scaled down,
3. the Solr async tracking entry for that backup is deleted ([DELETESTATUS API](https://github.com/apache/solr/blob/e0fe9619839f7ef2b43496104fa539a5b091db9a/solr/core/src/java/org/apache/solr/handler/admin/CollectionsHandler.java#L855-L930), [tracker deleteSingleAsyncId](https://github.com/apache/solr/blob/e0fe9619839f7ef2b43496104fa539a5b091db9a/solr/core/src/java/org/apache/solr/cloud/DistributedApiAsyncTracker.java#L284-L287)),
4. and the operator is scaled back up.
After that:
- the backup files already exist in Solr,
- `REQUESTSTATUS` for the backup request ID returns `notfound` ([REQUESTSTATUS API](https://github.com/apache/solr/blob/e0fe9619839f7ef2b43496104fa539a5b091db9a/solr/core/src/java/org/apache/solr/handler/admin/CollectionsHandler.java#L793-L853), [tracker getAsyncTaskRequestStatus](https://github.com/apache/solr/blob/e0fe9619839f7ef2b43496104fa539a5b091db9a/solr/core/src/java/org/apache/solr/cloud/DistributedApiAsyncTracker.java#L217-L270)),
- but the `SolrBackup` CR still stays in `inProgress=true`,
- and the operator keeps polling instead of reaching a terminal state ([backup reconcile loop](https://github.com/apache/solr-operator/blob/ca9d3c5c37a59f29570a6b49a8da5dc614aba75e/controllers/solrbackup_controller.go#L284-L306), [requeue after 5s](https://github.com/apache/solr-operator/blob/ca9d3c5c37a59f29570a6b49a8da5dc614aba75e/controllers/solrbackup_controller.go#L141-L143)).
# Environment
- macOS
- local `kind` cluster
- Kubernetes / kind node version: `v1.32.1`
- `solr-operator` version: `v0.10.0-orerekease`
- `solr-operator` built from `master` on March 22, 2026 (`ca9d3c5c37a59f29570a6b49a8da5dc614aba75e`)
- Solr version: `9.10.0`
# Steps to reproduce
1. Deploy `solr-operator` on a local `kind` cluster.
2. Create a 1-node `SolrCloud` with a local backup repository, then create a
collection and start a `SolrBackup` for it.
3. As soon as the backup first shows `inProgress=true`, scale the
`solr-operator` deployment down to `0`.
4. While the operator is down, wait for the Solr async request to finish, then
delete only that async status entry with `DELETESTATUS`.
5. Confirm the backup data still exists, but `REQUESTSTATUS` for that same
request ID now returns `notfound`.
6. Scale the operator back up to `1` and observe that the `SolrBackup` CR never
reaches a terminal state.
The stuck status looks like:
```yaml
status:
collectionBackupStatuses:
- asyncBackupStatus: notfound
inProgress: true
```
and it never sets `finished: true` or `successful: true`.
# Expected behavior
Once an accepted backup later becomes `notfound`, the operator should not leave the CR in `InProgress` forever.
It should eventually either:
- recover, or
- mark the backup failed with a clear reason.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [I] SolrBackup may never finish if REQUESTSTATUS returns notfound for an accepted backup [solr-operator]
Posted by "TravisFarrellMCC (via GitHub)" <gi...@apache.org>.
TravisFarrellMCC commented on issue #824:
URL: https://github.com/apache/solr-operator/issues/824#issuecomment-6332039095
If the fix provided by @autokctrl-team are valid, please merge his PR into the code. This is a serverly disruptive and critial issue for a large enterprise with a large and very busy Solr Cloud. I'm adding another production data point. This is hitting us repeatedly on recurring backups, and it's silent.
**Environment:** solr-operator v0.8.1, Solr 9.8.1, GKE, GCS backup repository, one recurring `SolrBackup` (`@hourly`, `maxSaved: 48`) over 41 collections, in two separate clusters (primary and DR).
**What we see over 11 days (Oct 10–21):**
- 7 separate stalls in the primary cluster. Each time, a subset of collections in one scheduled run (2 to 36 of 41) ended with `asyncBackupStatus: notfound` and `finished: null`, and the run never went terminal. Top-level `finished`/`successful` stayed `null`, so no further runs were scheduled.
- Gaps ranged from ~2.5 hours to ~92 hours. Only 48 runs completed in a window that should have had ~260.
- Our DR cluster stalled the same way after 9 hourly runs and stayed stalled until we noticed.
- In the retained history, `notfound` appears only on collections in stalled runs (74 of 2,009 collection statuses). It never appears on a collection that finished. In every case we checked, Solr had actually written the backup (`backup_N.properties` and shard metadata are present in GCS). Only the operator's tracking was lost.
**Why this matters:** nothing alerts. The operator reports no error, the CR just stays `inProgress`, and the backup schedule silently stops. We found out because we built external freshness monitoring on the GCS bucket. Before that, our longest gap was nearly four days without a backup.
We also confirmed that `controllers/util/backup_util.go` and `controllers/util/solr_api/backup.go` are unchanged between v0.8.1 and `main`, so upgrading to v0.9.1 doesn't help. This matches @benediktarnold's reports above on v0.9.1.
+1 for #846. Treating `notfound` as terminal and unsuccessful, while keeping the status visible, is the right behavior: a failed run is far better than a frozen schedule. Happy to test a build of #846 against our setup and report back.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [I] SolrBackup may never finish if REQUESTSTATUS returns notfound for an accepted backup [solr-operator]
Posted by "benediktarnold (via GitHub)" <gi...@apache.org>.
benediktarnold commented on issue #824:
URL: https://github.com/apache/solr-operator/issues/824#issuecomment-4857175120
Confirming this on the **latest release v0.9.1** (Solr 9.7, GCS repo), and adding
the impact on **recurring** backups, which is severe.
Our operator runs on GKE Autopilot and is restarted routinely (node
auto-provisioning/upgrades), so the precondition described here — backup accepted,
operator unavailable, async status later returns `notfound` — is easy to hit
without any manual intervention.
The key thing to add: for a **recurring** `SolrBackup`, a single stuck
`notfound`/`inProgress` collection wedges the *entire* recurrence, not just that
collection:
- top-level `finished` and `nextScheduled` stay `null`,
- the schedule never fires again and `maxSaved` pruning never runs,
- every other collection still reports `successful: true`, and no error surfaces.
Ours has been frozen at its original `startTimestamp`, with a few per-collection
statuses stuck in `notfound`/`inProgress`. The schedule was even changed during
that window with no effect, because the recurrence never advanced past the wedged
run.
Suggestion, in addition to making `notfound` terminal: compute
`finished`/`nextScheduled` from *terminal* per-collection states, so one stuck
collection can't halt the schedule and `maxSaved` retention indefinitely.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [I] SolrBackup may never finish if REQUESTSTATUS returns notfound for an accepted backup [solr-operator]
Posted by "Soumya8898 (via GitHub)" <gi...@apache.org>.
Soumya8898 commented on issue #824:
URL: https://github.com/apache/solr-operator/issues/824#issuecomment-4246016596
Hi @oyjhl , can you please assign this issue to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [I] SolrBackup may never finish if REQUESTSTATUS returns notfound for an accepted backup [solr-operator]
Posted by "autokctrl-team (via GitHub)" <gi...@apache.org>.
autokctrl-team commented on issue #824:
URL: https://github.com/apache/solr-operator/issues/824#issuecomment-5406119346
I tried to fix this issue and opened a PR. Please let me know if that fix makes sense to you. Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org