You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@solr.apache.org by GitBox <gi...@apache.org> on 2022/01/14 07:40:15 UTC
[GitHub] [solr-operator] hrvolapeter opened a new issue #393: GKE Autopilot constant re-deployment of pods
hrvolapeter opened a new issue #393:
URL: https://github.com/apache/solr-operator/issues/393
I'm runnig solr operator on GKE Autopilot clust with example from [here](https://solr.apache.org/operator/articles/explore-v030-gke.html) - the example requires to round-up resource requests to values supported by autopilot. Hovewer, operator keeps rededeploying deployments due to change in resource requsts, but the 'to' and 'from' valus are exactly the same. This bug is best observable with prometheus exporthes which is redeployed couple times per second, the same is happening with cluster but less often - seem like every couple minutes.
Logs, seem like relevant code is [this](https://github.com/apache/solr-operator/blob/94b62dc967ac4c6e1c8f16acf45d43298ca103d3/controllers/util/common.go#L545) - note the to and from clauses
```
controller-runtime.manager.controller.solrprometheusexporter Update required because field changed {"reconciler group": "solr.apache.org", "reconciler kind": "SolrPrometheusExporter", "name": "explore-prom-exporter", "namespace": "default", "deployment": "explore-prom-exporter-solr-metrics", "kind": "deployment", "field": "Spec.Template.Spec.Containers[0].Resources", "from": {"limits":{"cpu":"250m","ephemeral-storage":"1Gi","memory":"512Mi"},"requests":{"cpu":"250m","ephemeral-storage":"1Gi","memory":"512Mi"}}, "to": {"limits":{"cpu":"250m","ephemeral-storage":"1Gi","memory":"512Mi"},"requests":{"cpu":"250m","ephemeral-storage":"1Gi","memory":"512Mi"}}}
```
```
Error
2022-01-13T21:46:02.884596420Z2022-01-13T21:46:02.884Z INFO controller-runtime.manager.controller.solrcloud Update required because field changed {"reconciler group": "solr.apache.org", "reconciler kind": "SolrCloud", "name": "explore", "namespace": "default", "zookeeperCluster": "explore-solrcloud-zookeeper", "kind": "zookeeperCluster", "field": "Spec.Pod.Resources", "from": {"limits":{"cpu":"250m","ephemeral-storage":"512Mi","memory":"500Mi"},"requests":{"cpu":"250m","ephemeral-storage":"512Mi","memory":"500Mi"}}, "to": {"limits":{"cpu":"250m","ephemeral-storage":"512Mi","memory":"500Mi"},"requests":{"cpu":"250m","ephemeral-storage":"512Mi","memory":"500Mi"}}}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr-operator] HoustonPutman commented on issue #393: GKE Autopilot constant re-deployment of pods
Posted by GitBox <gi...@apache.org>.
HoustonPutman commented on issue #393:
URL: https://github.com/apache/solr-operator/issues/393#issuecomment-1016732811
Ok so given your logs, the first thing to fix is that the operator isn't treating the same resource amounts as actually equal. This isn't actually related to GKE Autopilot, but it should be fixing that part of your system. We will need to introduce a separate PR to not actually sync resources for the pods. (Note, I don't think this will work with the Zookeeper pods managed by the Zookeeper Operator).
To be clear, you are using the VerticalPodAutoscaler that is a part of the GKE Autopilot correct? That's what is changing the resource amounts for the pods?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr-operator] hrvolapeter commented on issue #393: GKE Autopilot constant re-deployment of pods
Posted by GitBox <gi...@apache.org>.
hrvolapeter commented on issue #393:
URL: https://github.com/apache/solr-operator/issues/393#issuecomment-1020441408
Hi @HoustonPutman thanks for taking a look.
I'm not using VerticalPodAutoscaler at least not knowingly, however the autopilot changing resources is based on [this](https://cloud.google.com/kubernetes-engine/docs/concepts/autopilot-overview#allowable_resource_ranges) documentation. Whenever the resources outside of limit / allowed combination autopilot changes it to the closest allowed value. It's definitely possible to figure right values on your own - but even then I hit the issues where equal diffs are not treated as equal
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr-operator] hrvolapeter commented on issue #393: GKE Autopilot constant re-deployment of pods
Posted by GitBox <gi...@apache.org>.
hrvolapeter commented on issue #393:
URL: https://github.com/apache/solr-operator/issues/393#issuecomment-1016629621
As far as I know only the resources are overwritten by Autopilot.
Autopilot requires requests / limits divisible by 512Mi for ram, and there are similar requirements for disk and cpu as well, however what I found more surprising even if you follow requirements the solr operator keeps redeploying the pods even if old and updated resource requests by autopilot are exactly the same. I've added the diff in the description
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr-operator] HoustonPutman commented on issue #393: GKE Autopilot constant re-deployment of pods
Posted by GitBox <gi...@apache.org>.
HoustonPutman commented on issue #393:
URL: https://github.com/apache/solr-operator/issues/393#issuecomment-1015584503
So the Autopilot re-writes the resources for the pod correct? Are there any other fields we should stop syncing/updating if autopilot is enabled?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr-operator] hrvolapeter commented on issue #393: GKE Autopilot constant re-deployment of pods
Posted by GitBox <gi...@apache.org>.
hrvolapeter commented on issue #393:
URL: https://github.com/apache/solr-operator/issues/393#issuecomment-1020441408
Hi @HoustonPutman thanks for taking a look.
I'm not using VerticalPodAutoscaler at least not knowingly, however the autopilot changing resources is based on [this](https://cloud.google.com/kubernetes-engine/docs/concepts/autopilot-overview#allowable_resource_ranges) documentation. Whenever the resources outside of limit / allowed combination autopilot changes it to the closest allowed value. It's definitely possible to figure right values on your own - but even then I hit the issues where equal diffs are not treated as equal
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org