You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@solr.apache.org by GitBox <gi...@apache.org> on 2022/12/16 13:12:48 UTC
[GitHub] [solr-operator] vipul-06 opened a new issue, #506: solr cloud backup gcs
vipul-06 opened a new issue, #506:
URL: https://github.com/apache/solr-operator/issues/506
I have created solrcloud using the yaml defined below
```
`apiVersion: solr.apache.org/v1beta1
kind: SolrCloud
metadata:
name: example
namespace: dev-backend
spec:
backupRepositories:
- name: "gcs-backups-1"
gcs:
bucket: "vipul-test-bucket" # Required
gcsCredentialSecret:
name: "newsecret"
key: "my-key.json"
replicas: 3
solrImage:
tag: 9.0.0`
```
I am taking backup of my solr collections in gcs bucket
My solr backup yaml is as below
```
apiVersion: solr.apache.org/v1beta1
kind: SolrBackup
metadata:
name: gcs-backup
namespace: dev-backend
spec:
solrCloud: example
repositoryName: "gcs-backups-1"
collections:
- demo
- test
- new
```
My backup is starting but not getting anything on my bucket it is empty.
Also my solrbackup is not getting completed it is as follows
```
NAME CLOUD STARTED FINISHED SUCCESSFUL NEXTBACKUP AGE
gcs-backup example 60m 60m
local-backup example 4h20m true true 4h20m
```
I firstly created a pvc backup it got completed but my gcs backup is started but not finishing also no data in my bucket
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr-operator] pbackup12345 commented on issue #506: solr cloud backup gcs
Posted by "pbackup12345 (via GitHub)" <gi...@apache.org>.
pbackup12345 commented on issue #506:
URL: https://github.com/apache/solr-operator/issues/506#issuecomment-1500885575
I encountered the same problem on GCS backup. I solved the first part as you said with first creating a path and then pointing the baseLocation there. Counter-intuitively it needs to be written without a preceding slash. The backup is now created (files and folders) but the backup still doesn't complete. It remains in a pending status indefinitely.
I see no errors in the log.
This is the describe result for the backup:
```(base) peterszeman@Peters-MacBook-Air solr % kubectl describe solrbackup local-backup9
Name: local-backup9
Namespace: sop030
Labels: <none>
Annotations: <none>
API Version: solr.apache.org/v1beta1
Kind: SolrBackup
Metadata:
Creation Timestamp: 2023-04-08T12:37:18Z
Generation: 1
Managed Fields:
API Version: solr.apache.org/v1beta1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:kubectl.kubernetes.io/last-applied-configuration:
f:spec:
.:
f:collections:
f:repositoryName:
f:solrCloud:
Manager: kubectl-client-side-apply
Operation: Update
Time: 2023-04-08T12:37:18Z
API Version: solr.apache.org/v1beta1
Fields Type: FieldsV1
fieldsV1:
f:status:
.:
f:collectionBackupStatuses:
f:solrVersion:
f:startTimestamp:
Manager: solr-operator
Operation: Update
Subresource: status
Time: 2023-04-08T12:37:18Z
Resource Version: 330672
UID: 23ea0f0b-0db7-412d-943f-e3ca58bda906
Spec:
Collections:
dsearch
Repository Name: gcs-backups-1
Solr Cloud: explore
Status:
Collection Backup Statuses:
Async Backup Status: notfound
Backup Name: local-backup9-dsearch
Collection: dsearch
In Progress: true
Start Timestamp: 2023-04-08T12:37:18Z
Solr Version: 8.11.0
Start Timestamp: 2023-04-08T12:37:18Z
Events: <none>```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr-operator] HoustonPutman commented on issue #506: solr cloud backup gcs
Posted by "HoustonPutman (via GitHub)" <gi...@apache.org>.
HoustonPutman commented on issue #506:
URL: https://github.com/apache/solr-operator/issues/506#issuecomment-1554854917
To sum up the issue, when using backups users sometimes have backups that succeed in Solr, but the SolrBackup status has the following two properties:
- `In Progress: true`
- `Async Backup Status: notfound`
From what I can discern, there are two possible reasons why the backup isn't being "finished" by the Solr Operator:
- The error was never handled when the backup was started. Thus an asyncId was never actually created, and why we get "notfound" when querying the status of the async command. (This can happen and needs to be fixed)
- The backup was finished and the Solr operator deleted the asyncId in Solr, however the backup status failed to update, and thus on the next iteration of the reconcile loop, it could not find the backup status anymore.
Given that the backup succeeded in Solr, the second option is more likely for the failures that we are seeing listed here.
There are issues with the status updates failing because of conflicts.
But this issue should happen much, much less starting in `v0.7.0` because of #544.
However, given that this is happening to users every single time, I am less confident.
This should happen sporadically, since its a race condition.
I will create a PR that starts to address these issues, so we can have y'all test it out and see what works.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr-operator] cesarfm commented on issue #506: solr cloud backup gcs
Posted by "cesarfm (via GitHub)" <gi...@apache.org>.
cesarfm commented on issue #506:
URL: https://github.com/apache/solr-operator/issues/506#issuecomment-1543779140
Hi, I am observing exactly the same symptoms described here, and in the same order (I am using GCS and first I was missing a `location`, after adding it I got stuck in the same next step as the OP). I believe this is the same issue than https://github.com/apache/solr-operator/issues/547 (which also fully matches with my situation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr-operator] HoustonPutman commented on issue #506: solr cloud backup gcs
Posted by GitBox <gi...@apache.org>.
HoustonPutman commented on issue #506:
URL: https://github.com/apache/solr-operator/issues/506#issuecomment-1363301872
As mentioned in your slack thread, you are seeing the following error:
```
ERROR controller-runtime.manager.controller.solrbackup Error while taking SolrCloud backup {"reconciler group": "[solr.apache.org](http://solr.apache.org/)", "reconciler kind": "SolrBackup", "name": "gcs-backup", "namespace": "dev-backend", "error": "Recieved bad response code of 500 from solr with response: {\n \"responseHeader\":{\n \"status\":500,\n \"QTime\":258},\n \"error\":{\n \"metadata\":[\n \"error-class\",\"org.apache.solr.common.SolrException\",\n \"root-error-class\",\"org.apache.solr.common.SolrException\"],\n \"msg\":\"specified location / does not exist.\",\n
```
The Backup command requires a location field, and the operator uses "/" as the default location. This works nicely with S3, since "/" can be used as the root node. With GCS "/" and "" are both valid starts to paths, so you would need to create the "/" path yourself.
Another option to move forward is to specify a real "location" in the backup, or the GCS repo spec, and manually create that path in GCS before starting everything.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org