You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@solr.apache.org by "brickpattern (via GitHub)" <gi...@apache.org> on 2023/12/14 22:34:01 UTC
[I] SolrCloud Pod moved to new Node - Replica Migration pending [solr-operator]
brickpattern opened a new issue, #668:
URL: https://github.com/apache/solr-operator/issues/668
Environment:
Solr Operator Helm : 0.8.0
Solr 9.4 container image
3 node cluster
Persistent storage option (w/ localvolume provisioner)
Managed upgrade strategy.
The K8S node for solrcloud-0 got cordon and Pod was moved to a new node.
When the pod came up on new node, its recognized by as part of the SolrCloud statefulset, but at the Collection level replica was lost on the node. Looking at the Stateful , there's a cluster lock.
solr.apache.org/clusterOpsLock: >-
{"operation":"RollingUpdate","lastStartTime":"2023-12-14T22:08:47Z","metadata":"{\"requiresReplicaMigration\":false}"}
**Should the operator automatically do the replica migration?**
I have read about Rebalance API and using 9.4 version.
**Is there a way to manually kick off the replica migration step to that specific POD?**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [I] SolrCloud Pod moved to new Node - Replica Migration pending [solr-operator]
Posted by "HoustonPutman (via GitHub)" <gi...@apache.org>.
HoustonPutman commented on issue #668:
URL: https://github.com/apache/solr-operator/issues/668#issuecomment-1921857745
Ahhh yes during node draining. That is a problem.
Yes, that is correct. What I would do is issue a Replace node command, moving all of the replicas off of the data-less pod. Then you can do a balance after that to move replicas back onto that pod.
It would be nice to have a command to fix all of the data on broken replicas. Maybe I'll make a JIRA for that.
One thing the operator can do is notice that a PV has changed, and if so automate the replica moving to restore data. Can you confirm that the PVs that are tied to the Solr PVCs change after draining the node? If so we can watch those PVs and try to fix the data if they are changed. (i.e. the data might be gone)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [I] SolrCloud Pod moved to new Node - Replica Migration pending [solr-operator]
Posted by "brickpattern (via GitHub)" <gi...@apache.org>.
brickpattern commented on issue #668:
URL: https://github.com/apache/solr-operator/issues/668#issuecomment-1861606488
changing the configs and eliminated the dependency on ZK on same name.
As well tested with both ephemeral and persistent storage. Regardless of the dataStorage, when the POD comes up ... the location its looking seems to be $SOLR_HOME/data/<<collection_shardN_replica_nN>>
When the POD comes up , there is no core.properties file.
Looking at other PODs, this file respective to that replica was written by "#Written by CorePropertiesLocator"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [I] SolrCloud Pod moved to new Node - Replica Migration pending [solr-operator]
Posted by "HoustonPutman (via GitHub)" <gi...@apache.org>.
HoustonPutman commented on issue #668:
URL: https://github.com/apache/solr-operator/issues/668#issuecomment-1915632109
Newer versions of Solr do not have an AutoAddReplica feature.
What kind of persistent volumes are you using? The data should not be missing when the pod is restarted. That's a failure of kubernetes/your PVC, and the Solr Operator isn't built to handle that. When you are running with Persistent Data it will expect the data to be there when restarted.
If you are running with ephemeral data, it will remove the data from the node before killing the pod. It can get into a bad state if the pod is killed on its own, and the data isn't moved beforehand.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [I] SolrCloud Pod moved to new Node - Replica Migration pending [solr-operator]
Posted by "brickpattern (via GitHub)" <gi...@apache.org>.
brickpattern commented on issue #668:
URL: https://github.com/apache/solr-operator/issues/668#issuecomment-1915170268
bump...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [I] SolrCloud Pod moved to new Node - Replica Migration pending [solr-operator]
Posted by "brickpattern (via GitHub)" <gi...@apache.org>.
brickpattern commented on issue #668:
URL: https://github.com/apache/solr-operator/issues/668#issuecomment-1856850726
Solr Operator logs
```
2023-12-14T21:39:42Z INFO Update required because field changed {"controller": "solrcloud", "controllerGroup": "solr.apache.org", "controllerKind": "SolrCloud", "SolrCloud": {"name":"solr","namespace":"solr"}, "namespace": "solr", "name": "solr", "reconcileID": "c747c5ad-f7b1-400a-8957-ff60a71b7531", "statefulSet": "solr-solrcloud", "kind": "statefulSet", "field": "Spec.Template.Annotations", "from": {"kubectl.kubernetes.io/restartedAt":"2023-12-14T15:39:42-06:00","solr.apache.org/solrXmlMd5":"5fe99d590bc63efc3caa743ca939aa5a"}, "to": {"solr.apache.org/solrXmlMd5":"5fe99d590bc63efc3caa743ca939aa5a"}}
2023-12-14T21:39:42Z INFO Updating StatefulSet {"controller": "solrcloud", "controllerGroup": "solr.apache.org", "controllerKind": "SolrCloud", "SolrCloud": {"name":"solr","namespace":"solr"}, "namespace": "solr", "name": "solr", "reconcileID": "c747c5ad-f7b1-400a-8957-ff60a71b7531", "statefulSet": "solr-solrcloud"}
2023-12-14T21:39:42Z INFO Started locked clusterOp {"controller": "solrcloud", "controllerGroup": "solr.apache.org", "controllerKind": "SolrCloud", "SolrCloud": {"name":"solr","namespace":"solr"}, "namespace": "solr", "name": "solr", "reconcileID": "122fddd5-d036-4d31-872e-0d4b95bb27a3", "clusterOp": "RollingUpdate", "clusterOpMetadata": "{\"requiresReplicaMigration\":false}"}
2023-12-14T21:39:42Z INFO Updating SolrCloud Status {"controller": "solrcloud", "controllerGroup": "solr.apache.org", "controllerKind": "SolrCloud", "SolrCloud": {"name":"solr","namespace":"solr"}, "namespace": "solr", "name": "solr", "reconcileID": "122fddd5-d036-4d31-872e-0d4b95bb27a3", "status": {"solrNodes":[{"name":"solr-solrcloud-0","nodeName":"ip-x-y-162-17.us-west-2.compute.internal","internalAddress":"http://solr-solrcloud-0.solr","externalAddress":"http://solr-solr-solrcloud-0.k8s.solr.cloud","ready":true,"version":"0.8","specUpToDate":false,"scheduledForDeletion":false},{"name":"solr-solrcloud-1","nodeName":"ip-x-y-160-139.us-west-2.compute.internal","internalAddress":"http://solr-solrcloud-1.solr","externalAddress":"http://solr-solr-solrcloud-1.k8s.solr.cloud","ready":true,"version":"0.8","specUpToDate":false,"scheduledForDeletion":false},{"name":"solr-solrcloud-2","nodeName":"ip-x-y-163-213.us-west-2.compute.internal","internalAddress":"http://solr-solrcloud-2.solr","
externalAddress":"http://solr-solr-solrcloud-2.k8s.solr.cloud","ready":true,"version":"0.8","specUpToDate":false,"scheduledForDeletion":false}],"replicas":3,"podSelector":"solr-cloud=solr,technology=solr-cloud","readyReplicas":3,"upToDateNodes":0,"version":"0.8","internalCommonAddress":"http://solr-solrcloud-common.solr","externalCommonAddress":"http://solr-solr-solrcloud.k8s.solr.cloud","zookeeperConnectionInfo":{"internalConnectionString":"solr-solrcloud-zookeeper-0.solr-solrcloud-zookeeper-headless.solr.svc.cluster.local:2181,solr-solrcloud-zookeeper-1.solr-solrcloud-zookeeper-headless.solr.svc.cluster.local:2181,solr-solrcloud-zookeeper-2.solr-solrcloud-zookeeper-headless.solr.svc.cluster.local:2181","externalConnectionString":"N/A","chroot":"/"},"backupRestoreReady":false}}
2023-12-14T21:39:42Z INFO Removed unneeded clusterOpLock annotation from statefulSet {"controller": "solrcloud", "controllerGroup": "solr.apache.org", "controllerKind": "SolrCloud", "SolrCloud": {"name":"solr","namespace":"solr"}, "namespace": "solr", "name": "solr", "reconcileID": "cb77fc36-beda-4e54-a0c5-0b2726756c66", "reason": "RollingUpdate complete"}
2023-12-14T21:39:42Z INFO Updating SolrCloud Status {"controller": "solrcloud", "controllerGroup": "solr.apache.org", "controllerKind": "SolrCloud", "SolrCloud": {"name":"solr","namespace":"solr"}, "namespace": "solr", "name": "solr", "reconcileID": "cb77fc36-beda-4e54-a0c5-0b2726756c66", "status": {"solrNodes":[{"name":"solr-solrcloud-0","nodeName":"ip-x-y-162-17.us-west-2.compute.internal","internalAddress":"http://solr-solrcloud-0.solr","externalAddress":"http://solr-solr-solrcloud-0.k8s.solr.cloud","ready":true,"version":"0.8","specUpToDate":true,"scheduledForDeletion":false},{"name":"solr-solrcloud-1","nodeName":"ip-x-y-160-139.us-west-2.compute.internal","internalAddress":"http://solr-solrcloud-1.solr","externalAddress":"http://solr-solr-solrcloud-1.k8s.solr.cloud","ready":true,"version":"0.8","specUpToDate":true,"scheduledForDeletion":false},{"name":"solr-solrcloud-2","nodeName":"ip-x-y-163-213.us-west-2.compute.internal","internalAddress":"http://solr-solrcloud-2.solr","ex
ternalAddress":"http://solr-solr-solrcloud-2.k8s.solr.cloud","ready":true,"version":"0.8","specUpToDate":true,"scheduledForDeletion":false}],"replicas":3,"podSelector":"solr-cloud=solr,technology=solr-cloud","readyReplicas":3,"upToDateNodes":3,"version":"0.8","internalCommonAddress":"http://solr-solrcloud-common.solr","externalCommonAddress":"http://solr-solr-solrcloud.k8s.solr.cloud","zookeeperConnectionInfo":{"internalConnectionString":"solr-solrcloud-zookeeper-0.solr-solrcloud-zookeeper-headless.solr.svc.cluster.local:2181,solr-solrcloud-zookeeper-1.solr-solrcloud-zookeeper-headless.solr.svc.cluster.local:2181,solr-solrcloud-zookeeper-2.solr-solrcloud-zookeeper-headless.solr.svc.cluster.local:2181","externalConnectionString":"N/A","chroot":"/"},"backupRestoreReady":false}}
2023-12-14T21:58:46Z INFO Update required because field changed {"controller": "solrcloud", "controllerGroup": "solr.apache.org", "controllerKind": "SolrCloud", "SolrCloud": {"name":"solr","namespace":"solr"}, "namespace": "solr", "name": "solr", "reconcileID": "71340161-9ee5-47d9-b7f4-7630ad2ab132", "statefulSet": "solr-solrcloud", "kind": "statefulSet", "field": "Spec.Template.Annotations", "from": {"solr.apache.org/solrXmlMd5":"5fe99d590bc63efc3caa743ca939aa5a"}, "to": {"manualrestart":"2023-12-14T00:00:01Z","solr.apache.org/solrXmlMd5":"5fe99d590bc63efc3caa743ca939aa5a"}}
2023-12-14T21:58:46Z INFO Updating StatefulSet {"controller": "solrcloud", "controllerGroup": "solr.apache.org", "controllerKind": "SolrCloud", "SolrCloud": {"name":"solr","namespace":"solr"}, "namespace": "solr", "name": "solr", "reconcileID": "71340161-9ee5-47d9-b7f4-7630ad2ab132", "statefulSet": "solr-solrcloud"}
2023-12-14T21:58:46Z INFO Started locked clusterOp {"controller": "solrcloud", "controllerGroup": "solr.apache.org", "controllerKind": "SolrCloud", "SolrCloud": {"name":"solr","namespace":"solr"}, "namespace": "solr", "name": "solr", "reconcileID": "a09d90c0-06f2-4b59-9c68-463c97261d5b", "clusterOp": "RollingUpdate", "clusterOpMetadata": "{\"requiresReplicaMigration\":false}"}
2023-12-14T21:58:46Z INFO Updating SolrCloud Status {"controller": "solrcloud", "controllerGroup": "solr.apache.org", "controllerKind": "SolrCloud", "SolrCloud": {"name":"solr","namespace":"solr"}, "namespace": "solr", "name": "solr", "reconcileID": "a09d90c0-06f2-4b59-9c68-463c97261d5b", "status": {"solrNodes":[{"name":"solr-solrcloud-0","nodeName":"ip-x-y-162-17.us-west-2.compute.internal","internalAddress":"http://solr-solrcloud-0.solr","externalAddress":"http://solr-solr-solrcloud-0.k8s.solr.cloud","ready":true,"version":"0.8","specUpToDate":false,"scheduledForDeletion":false},{"name":"solr-solrcloud-1","nodeName":"ip-x-y-160-139.us-west-2.compute.internal","internalAddress":"http://solr-solrcloud-1.solr","externalAddress":"http://solr-solr-solrcloud-1.k8s.solr.cloud","ready":true,"version":"0.8","specUpToDate":false,"scheduledForDeletion":false},{"name":"solr-solrcloud-2","nodeName":"ip-x-y-163-213.us-west-2.compute.internal","internalAddress":"http://solr-solrcloud-2.solr","
externalAddress":"http://solr-solr-solrcloud-2.k8s.solr.cloud","ready":true,"version":"0.8","specUpToDate":false,"scheduledForDeletion":false}],"replicas":3,"podSelector":"solr-cloud=solr,technology=solr-cloud","readyReplicas":3,"upToDateNodes":0,"version":"0.8","internalCommonAddress":"http://solr-solrcloud-common.solr","externalCommonAddress":"http://solr-solr-solrcloud.k8s.solr.cloud","zookeeperConnectionInfo":{"internalConnectionString":"solr-solrcloud-zookeeper-0.solr-solrcloud-zookeeper-headless.solr.svc.cluster.local:2181,solr-solrcloud-zookeeper-1.solr-solrcloud-zookeeper-headless.solr.svc.cluster.local:2181,solr-solrcloud-zookeeper-2.solr-solrcloud-zookeeper-headless.solr.svc.cluster.local:2181","externalConnectionString":"N/A","chroot":"/"},"backupRestoreReady":false}}
2023-12-14T21:58:46Z INFO ManagedUpdateSelector Pod update selection started. {"controller": "solrcloud", "controllerGroup": "solr.apache.org", "controllerKind": "SolrCloud", "SolrCloud": {"name":"solr","namespace":"solr"}, "namespace": "solr", "name": "solr", "reconcileID": "22e68695-cbc6-4f0d-8a40-7cf26a1841b6", "outOfDatePods": 3, "maxPodsUnavailable": 1, "unavailableUpdatedPods": 0, "outOfDatePodsNotStarted": 0, "alreadyScheduledForDeletion": 0, "maxPodsToUpdate": 1}
2023-12-14T21:58:46Z INFO ManagedUpdateSelector Pod selected to be deleted for update. {"controller": "solrcloud", "controllerGroup": "solr.apache.org", "controllerKind": "SolrCloud", "SolrCloud": {"name":"solr","namespace":"solr"}, "namespace": "solr", "name": "solr", "reconcileID": "22e68695-cbc6-4f0d-8a40-7cf26a1841b6", "pod": "solr-solrcloud-0", "reason": "Pod's replicas are safe to take down, adhering to the minimum active replicas per shard."}
2023-12-14T21:58:46Z INFO ManagedUpdateSelector Pod update selection complete. Maximum number of pods able to be updated reached. {"controller": "solrcloud", "controllerGroup": "solr.apache.org", "controllerKind": "SolrCloud", "SolrCloud": {"name":"solr","namespace":"solr"}, "namespace": "solr", "name": "solr", "reconcileID": "22e68695-cbc6-4f0d-8a40-7cf26a1841b6", "maxPodsToUpdate": 1}
2023-12-14T21:58:46Z INFO ManagedUpdateSelector Deleting solr pod for update {"controller": "solrcloud", "controllerGroup": "solr.apache.org", "controllerKind": "SolrCloud", "SolrCloud": {"name":"solr","namespace":"solr"}, "namespace": "solr", "name": "solr", "reconcileID": "22e68695-cbc6-4f0d-8a40-7cf26a1841b6", "pod": "solr-solrcloud-0"}
2023-12-14T21:58:46Z INFO Updating SolrCloud Status {"controller": "solrcloud", "controllerGroup": "solr.apache.org", "controllerKind": "SolrCloud", "SolrCloud": {"name":"solr","namespace":"solr"}, "namespace": "solr", "name": "solr", "reconcileID": "22e68695-cbc6-4f0d-8a40-7cf26a1841b6", "status": {"solrNodes":[{"name":"solr-solrcloud-0","nodeName":"ip-x-y-162-17.us-west-2.compute.internal","internalAddress":"http://solr-solrcloud-0.solr","externalAddress":"http://solr-solr-solrcloud-0.k8s.solr.cloud","ready":true,"version":"0.8","specUpToDate":false,"scheduledForDeletion":false},{"name":"solr-solrcloud-1","nodeName":"ip-x-y-160-139.us-west-2.compute.internal","internalAddress":"http://solr-solrcloud-1.solr","externalAddress":"http://solr-solr-solrcloud-1.k8s.solr.cloud","ready":true,"version":"0.8","specUpToDate":false,"scheduledForDeletion":false},{"name":"solr-solrcloud-2","nodeName":"ip-x-y-163-213.us-west-2.compute.internal","internalAddress":"http://solr-solrcloud-2.solr","
externalAddress":"http://solr-solr-solrcloud-2.k8s.solr.cloud","ready":true,"version":"0.8","specUpToDate":false,"scheduledForDeletion":false}],"replicas":3,"podSelector":"solr-cloud=solr,technology=solr-cloud","readyReplicas":3,"upToDateNodes":0,"version":"0.8","internalCommonAddress":"http://solr-solrcloud-common.solr","externalCommonAddress":"http://solr-solr-solrcloud.k8s.solr.cloud","zookeeperConnectionInfo":{"internalConnectionString":"solr-solrcloud-zookeeper-0.solr-solrcloud-zookeeper-headless.solr.svc.cluster.local:2181,solr-solrcloud-zookeeper-1.solr-solrcloud-zookeeper-headless.solr.svc.cluster.local:2181,solr-solrcloud-zookeeper-2.solr-solrcloud-zookeeper-headless.solr.svc.cluster.local:2181","externalConnectionString":"N/A","chroot":"/"},"backupRestoreReady":false}}
2023-12-14T21:58:46Z INFO ManagedUpdateSelector Pod update selection not started. The number of unavailable pods unavailable (or scheduled for deletion) equals or exceeds the calculated maxPodsUnavailable. {"controller": "solrcloud", "controllerGroup": "solr.apache.org", "controllerKind": "SolrCloud", "SolrCloud": {"name":"solr","namespace":"solr"}, "namespace": "solr", "name": "solr", "reconcileID": "d091496b-8199-419f-b541-3709fc4cbd03", "outOfDatePods": 2, "maxPodsUnavailable": 1, "unavailableUpdatedPods": 0, "outOfDatePodsNotStarted": 0, "alreadyScheduledForDeletion": 1}
2023-12-14T21:58:46Z INFO ManagedUpdateSelector Deleting solr pod for update {"controller": "solrcloud", "controllerGroup": "solr.apache.org", "controllerKind": "SolrCloud", "SolrCloud": {"name":"solr","namespace":"solr"}, "namespace": "solr", "name": "solr", "reconcileID": "d091496b-8199-419f-b541-3709fc4cbd03", "pod": "solr-solrcloud-0"}
2023-12-14T21:58:46Z INFO Updating SolrCloud Status {"controller": "solrcloud", "controllerGroup": "solr.apache.org", "controllerKind": "SolrCloud", "SolrCloud": {"name":"solr","namespace":"solr"}, "namespace": "solr", "name": "solr", "reconcileID": "d091496b-8199-419f-b541-3709fc4cbd03", "status": {"solrNodes":[{"name":"solr-solrcloud-0","nodeName":"ip-x-y-162-17.us-west-2.compute.internal","internalAddress":"http://solr-solrcloud-0.solr","externalAddress":"http://solr-solr-solrcloud-0.k8s.solr.cloud","ready":false,"version":"0.8","specUpToDate":false,"scheduledForDeletion":true},{"name":"solr-solrcloud-1","nodeName":"ip-x-y-160-139.us-west-2.compute.internal","internalAddress":"http://solr-solrcloud-1.solr","externalAddress":"http://solr-solr-solrcloud-1.k8s.solr.cloud","ready":true,"version":"0.8","specUpToDate":false,"scheduledForDeletion":false},{"name":"solr-solrcloud-2","nodeName":"ip-x-y-163-213.us-west-2.compute.internal","internalAddress":"http://solr-solrcloud-2.solr","
externalAddress":"http://solr-solr-solrcloud-2.k8s.solr.cloud","ready":true,"version":"0.8","specUpToDate":false,"scheduledForDeletion":false}],"replicas":3,"podSelector":"solr-cloud=solr,technology=solr-cloud","readyReplicas":2,"upToDateNodes":0,"version":"0.8","internalCommonAddress":"http://solr-solrcloud-common.solr","externalCommonAddress":"http://solr-solr-solrcloud.k8s.solr.cloud","zookeeperConnectionInfo":{"internalConnectionString":"solr-solrcloud-zookeeper-0.solr-solrcloud-zookeeper-headless.solr.svc.cluster.local:2181,solr-solrcloud-zookeeper-1.solr-solrcloud-zookeeper-headless.solr.svc.cluster.local:2181,solr-solrcloud-zookeeper-2.solr-solrcloud-zookeeper-headless.solr.svc.cluster.local:2181","externalConnectionString":"N/A","chroot":"/"},"backupRestoreReady":false}}
2023-12-14T21:58:47Z INFO ManagedUpdateSelector Pod update selection not started. The number of unavailable pods unavailable (or scheduled for deletion) equals or exceeds the calculated maxPodsUnavailable. {"controller": "solrcloud", "controllerGroup": "solr.apache.org", "controllerKind": "SolrCloud", "SolrCloud": {"name":"solr","namespace":"solr"}, "namespace": "solr", "name": "solr", "reconcileID": "63e8cd10-0938-451f-bdfe-d5c7e0bbcb6a", "outOfDatePods": 2, "maxPodsUnavailable": 1, "unavailableUpdatedPods": 0, "outOfDatePodsNotStarted": 0, "alreadyScheduledForDeletion": 1}
2023-12-14T21:58:47Z INFO ManagedUpdateSelector Deleting solr pod for update {"controller": "solrcloud", "controllerGroup": "solr.apache.org", "controllerKind": "SolrCloud", "SolrCloud": {"name":"solr","namespace":"solr"}, "namespace": "solr", "name": "solr", "reconcileID": "63e8cd10-0938-451f-bdfe-d5c7e0bbcb6a", "pod": "solr-solrcloud-0"}
2023-12-14T21:58:47Z INFO ManagedUpdateSelector Pod update selection not started. The number of unavailable pods unavailable (or scheduled for deletion) equals or exceeds the calculated maxPodsUnavailable. {"controller": "solrcloud", "controllerGroup": "solr.apache.org", "controllerKind": "SolrCloud", "SolrCloud": {"name":"solr","namespace":"solr"}, "namespace": "solr", "name": "solr", "reconcileID": "d7ad8183-8e6b-4377-8901-fbd4a8129672", "outOfDatePods": 2, "maxPodsUnavailable": 1, "unavailableUpdatedPods": 0, "outOfDatePodsNotStarted": 0, "alreadyScheduledForDeletion": 1}
2023-12-14T21:58:47Z INFO ManagedUpdateSelector Deleting solr pod for update {"controller": "solrcloud", "controllerGroup": "solr.apache.org", "controllerKind": "SolrCloud", "SolrCloud": {"name":"solr","namespace":"solr"}, "namespace": "solr", "name": "solr", "reconcileID": "d7ad8183-8e6b-4377-8901-fbd4a8129672", "pod": "solr-solrcloud-0"}
2023-12-14T21:58:49Z INFO ManagedUpdateSelector Pod update selection not started. The number of unavailable pods unavailable (or scheduled for deletion) equals or exceeds the calculated maxPodsUnavailable. {"controller": "solrcloud", "controllerGroup": "solr.apache.org", "controllerKind": "SolrCloud", "SolrCloud": {"name":"solr","namespace":"solr"}, "namespace": "solr", "name": "solr", "reconcileID": "291ef125-ec68-40aa-945f-c3fd10cfbf5b", "outOfDatePods": 2, "maxPodsUnavailable": 1, "unavailableUpdatedPods": 1, "outOfDatePodsNotStarted": 0, "alreadyScheduledForDeletion": 0}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [I] SolrCloud Pod moved to new Node - Replica Migration pending [solr-operator]
Posted by "janhoy (via GitHub)" <gi...@apache.org>.
janhoy commented on issue #668:
URL: https://github.com/apache/solr-operator/issues/668#issuecomment-6478410734
Closing in favour of #787, which covers the generalisable half of this: replicas stuck permanently DOWN after a pod comes back with an empty volume, and what the operator could do to recover them automatically.
For the immediate situation, the manual fix is `DELETEREPLICA` followed by `ADDREPLICA` on the affected replicas (or a node REPLACE then a balance). Local-volume PVs are node-bound by design, so a drained node means the data is genuinely gone and the operator has no way to move it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [I] SolrCloud Pod moved to new Node - Replica Migration pending [solr-operator]
Posted by "brickpattern (via GitHub)" <gi...@apache.org>.
brickpattern commented on issue #668:
URL: https://github.com/apache/solr-operator/issues/668#issuecomment-1915641211
I have tested using both Persistent and Ephemeral resulting in loss of data for that solr node.
For Persistent storage , using the local volume provisioner. As long as the POD comes back in the same EKS node it binds to the PVC to the same PV and retains the data. But when the POD gets scheduled to another EKS node (which is my scenario) the data is lost. the folders/directory for the core config on that replaced POD is void of any data.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [I] SolrCloud Pod moved to new Node - Replica Migration pending [solr-operator]
Posted by "janhoy (via GitHub)" <gi...@apache.org>.
janhoy closed issue #668: SolrCloud Pod moved to new Node - Replica Migration pending
URL: https://github.com/apache/solr-operator/issues/668
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [I] SolrCloud Pod moved to new Node - Replica Migration pending [solr-operator]
Posted by "HoustonPutman (via GitHub)" <gi...@apache.org>.
HoustonPutman commented on issue #668:
URL: https://github.com/apache/solr-operator/issues/668#issuecomment-1915739941
The only ways that local volumes work as PVs is if the PVs that are created have node limitations (i.e. the Pod connected to the PV cannot be rescheduled onto another node). Are you sure that the local volume provisioner is setup correctly?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [I] SolrCloud Pod moved to new Node - Replica Migration pending [solr-operator]
Posted by "brickpattern (via GitHub)" <gi...@apache.org>.
brickpattern commented on issue #668:
URL: https://github.com/apache/solr-operator/issues/668#issuecomment-1920076595
yes, PVs are setup correctly. Solr PODS come up correctly either by evicting or restarting.
The specific scenario I'm certifying is EKS node taint n drain ( replace node with new hardware).
So it appears from your description, Operator will NOT move the data as the local volume is tied to a EKs node.
Is there a recommendation to manually trigger data replication from other 2 nodes ? like a API call to fulfill the data
$SOLR_HOME/data/<<collection_shardN_replica_nN>>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [I] SolrCloud Pod moved to new Node - Replica Migration pending [solr-operator]
Posted by "brickpattern (via GitHub)" <gi...@apache.org>.
brickpattern commented on issue #668:
URL: https://github.com/apache/solr-operator/issues/668#issuecomment-1858507746
after redoing the whole exercise... i noticed... the pod that is being reprovisioned to new node came up with
<<collection_name>>_shard1_replica_n4
It bumped the replica # by +1 ... it should have been shard1_replica_n3
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org