You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@solr.apache.org by "ozlerhakan (via GitHub)" <gi...@apache.org> on 2023/09/08 14:36:18 UTC
[GitHub] [solr-operator] ozlerhakan opened a new issue, #618: Role Assignment for SOLR Replicas
ozlerhakan opened a new issue, #618:
URL: https://github.com/apache/solr-operator/issues/618
Hi Team!
I was wondering if we have possibility to assign different roles for replicas, each of which is essentially a SOLR node, using SOLR's [Node Roles](https://solr.apache.org/guide/solr/latest/deployment-guide/node-roles.html) feature?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [I] Role Assignment for SOLR Replicas [solr-operator]
Posted by "admintimcools (via GitHub)" <gi...@apache.org>.
admintimcools commented on issue #618:
URL: https://github.com/apache/solr-operator/issues/618#issuecomment-2325850651
Having this would really help our scenario as well. We have very heavy indexing which requires complete different cpu and ram compared to our search. Having the ability to spin up ingestion nodes and search nodes with different specs would be really helpful @HoustonPutman
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [I] Role Assignment for SOLR Replicas [solr-operator]
Posted by "HoustonPutman (via GitHub)" <gi...@apache.org>.
HoustonPutman commented on issue #618:
URL: https://github.com/apache/solr-operator/issues/618#issuecomment-1755683141
Maybe for v0.9.0. Using the roles API, it would be more feasible. We already do some defaulting when Pods are created (no API calls though). So it is feasible to add another defaulting method, which works with a readinessCondition.
Basically when a pod is started, the `nodeRoleInitialized` readinessCondition will be empty. The operator would have code to say, while the Solr container is ready and the `nodeRoleInitialized` readinessCondition is empty, call the roles API. When the API call is successful, set the `nodeRoleInitialized` readinessCondition to true. That way we have a stateful and idempotent way of handling the API calls.
How do you imagine this would be integrated into the SolrCloud CRD?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr-operator] HoustonPutman commented on issue #618: Role Assignment for SOLR Replicas
Posted by "HoustonPutman (via GitHub)" <gi...@apache.org>.
HoustonPutman commented on issue #618:
URL: https://github.com/apache/solr-operator/issues/618#issuecomment-1711811235
This is a very good question, but ultimately it would be a fairly big change to the way that the operator works.
Currently the Solr Operator uses a single StatefulSet to manage the pods for each SolrCloud instance. The statefulSet has a single pod spec, so pods are meant to be identical. That means we can't give different envVars (to set the node role) to different pods.
One option would be to have the Solr Operator manage a solr cloud in multiple StatefulSets, so that each statefulSet could have a different podSpec to . This is definitely something that could be done, but would be a fairly big undertaking. If you list your use case, and how you imagine you would want to configure the node roles, then it's something we could think about for `v0.9.0` or `v1.0.0`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr-operator] ozlerhakan commented on issue #618: Role Assignment for SOLR Replicas
Posted by "ozlerhakan (via GitHub)" <gi...@apache.org>.
ozlerhakan commented on issue #618:
URL: https://github.com/apache/solr-operator/issues/618#issuecomment-1716311937
Thank you @janhoy and @HoustonPutman for your interest in this question.
The primarily idea is to create a SolrCloud environment where we maintain a large dataset under 3+ distributed Solr instances with the backup-restore feature at hand. This mechanism is experimental and we conduct extensive tests on it. Additionally, I want to experiment with another scenario where shard-leaders and replicas reside in separate and isolated instances within the cluster. In this setup, leader-shards are responsible for updates and backup-restore tasks, while replicas can handle heavy users requests. However, I'm still unsure if this is the best practice and feasible when it comes to the SolrCloud concept.
Returning to my initial question, although my understanding of how the logic behind assigning node roles works is not solid at the moment, I was expecting a Solr API for nodes to assign new roles to running instances in the cluster at runtime as there is a Roles [API](https://solr.apache.org/guide/solr/latest/deployment-guide/node-roles.html#roles-api) that allows us to list some node level details at cluster level.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr-operator] janhoy commented on issue #618: Role Assignment for SOLR Replicas
Posted by "janhoy (via GitHub)" <gi...@apache.org>.
janhoy commented on issue #618:
URL: https://github.com/apache/solr-operator/issues/618#issuecomment-1716596075
I believe the role assignment is a startup property on purpose, designed to be a more static feature of a node that you don't want to be messed with at runtime. It would also be much more complex to, say, remove the "data" role from a node in-flight, would it then instantly evict all its cores, etc?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr-operator] janhoy commented on issue #618: Role Assignment for SOLR Replicas
Posted by "janhoy (via GitHub)" <gi...@apache.org>.
janhoy commented on issue #618:
URL: https://github.com/apache/solr-operator/issues/618#issuecomment-1711915485
In SIP-14 we discuss adding a `zookeeper` noe-role. Then it would make sense to start a separate statefulSet for the zk pods. They may need different role, different RAM, disk, CPU limit etc. The same could be true for other node-roles such as "data", "coordinator" and future roles. So perhaps if the operator had a way to configure optional group of pods with different combination of node-roles, and each would get its own statefulset. You could then have a group of pods which are optimized for querying (pull replica) etc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr-operator] ozlerhakan commented on issue #618: Role Assignment for SOLR Replicas
Posted by "ozlerhakan (via GitHub)" <gi...@apache.org>.
ozlerhakan commented on issue #618:
URL: https://github.com/apache/solr-operator/issues/618#issuecomment-1718053738
Indeed, It looks like a long and back and forth discussion, and I also noticed some of you shared my opinion. Well, at this point, it seems you have already made a conclusion about this SIP, what is done is already done. However, It would be excellent if you consider another option for defining an instance's role next to envVars in the future.
Regarding the main question, we'll likely have to rely on the operator to have support for specific envVars declarations in the `solrOpts` field within the `SolrCloud` CRD spec. Perhaps, the operator could introduce a new way to label parameters with the use of a mix of strings --`<metadata.name>` + `<solrcloud-kind-name>` + `<instance-number>`--. Therefore, we can assign specific flags to individual instances. For instance, if `metadata.name:test` and the cluster has 2 replicas/instances, they can be defined as `test-solrcloud-0:-Dsolr.node.roles=coordinator:on,data:off test-solrcloud-1:-Dsolr.node.roles=coordinator:off,data:on` It is just my first impression and it could also be done in another way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [I] Role Assignment for SOLR Replicas [solr-operator]
Posted by "HoustonPutman (via GitHub)" <gi...@apache.org>.
HoustonPutman commented on issue #618:
URL: https://github.com/apache/solr-operator/issues/618#issuecomment-6474511549
@msobkowiak-olx I do ultimately think this is the right way to go about it, and something that should be supported in the operator.
Sorry that you didn't get a reply when you posted a year ago. I think this would get support if y'all put something up now. (And it's very unfortunate that this moved you away from the operator, so it would be great to get support in. Managing Solr without the operator sounds very un-fun!)
Also I don't think it _has_ to be a breaking change. We can do node-type suffixes if the user has requested for node-types, and otherwise default to the regular suffix-less names. (This is less nice, but not too bad)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr-operator] ozlerhakan commented on issue #618: Role Assignment for SOLR Replicas
Posted by "ozlerhakan (via GitHub)" <gi...@apache.org>.
ozlerhakan commented on issue #618:
URL: https://github.com/apache/solr-operator/issues/618#issuecomment-1717014606
> designed to be a more static feature of a node that you don't want to be messed with at runtime.
Why would someone mess things up if they have the right API calls to arrange their cluster's nodes roles before populating any data and releasing it to production? Apart from this, one could also mess things up with some of the existing Solr APIs too. For instance, the Restore API lets us restore a backup even if there is a collection with the same name in the cluster. My point is, depending on how we approach, you might need to do some pre-checks before messing the whole cluster.
> would it then instantly evict all its cores, etc?
So what happens "if I restart a `data:on` Solr instance that holds data, but is now `data:off`?"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [I] Role Assignment for SOLR Replicas [solr-operator]
Posted by "richardbadman (via GitHub)" <gi...@apache.org>.
richardbadman commented on issue #618:
URL: https://github.com/apache/solr-operator/issues/618#issuecomment-6471286777
Hey, I know this is a bit of an old issue. But we've revisited using the solr operator again in our company. In our bare metal environments for each cluster we have typically;
- X number of machines for Solr only. 95% of these instances are typical solr cloud instances, and the remaining 5% are dedicated coordinator instances.
- 3 number of machines for ZK and a small solr instance for solr overseer.
I was curious how I would go about this when using the solr operator, and came across this issue. The way I have gone around this (as I couldn't see anything in the change log about this) was having 3 manifests, 1 for the main bulk of the cluster which was the data nodes and the zookeeper creation & management, the other two are almost identical, but differentiate their roles of coordinator and overseer with differing resource allocations. These two also referenced the zookeeper ensemble in the first manifest, and it seems to have worked well.
More than happy to provide a code example if needed. It might not be the _best_ solution, but it is one that seemed trivial and worked.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [I] Role Assignment for SOLR Replicas [solr-operator]
Posted by "msobkowiak-olx (via GitHub)" <gi...@apache.org>.
msobkowiak-olx commented on issue #618:
URL: https://github.com/apache/solr-operator/issues/618#issuecomment-6472954782
We solved it using our custom chart eventually (we opted-out from the operator, just used bare StatefulSets)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr-operator] janhoy commented on issue #618: Role Assignment for SOLR Replicas
Posted by "janhoy (via GitHub)" <gi...@apache.org>.
janhoy commented on issue #618:
URL: https://github.com/apache/solr-operator/issues/618#issuecomment-1717111620
There was a fairly thorough discussion on the design here https://lists.apache.org/thread/39q48fbtcxcl9btg24twj9bwowlro16d - perhaps you find the answer to the particular decision.
All in all, would be great to have support in the operator!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [I] Role Assignment for SOLR Replicas [solr-operator]
Posted by "ozlerhakan (via GitHub)" <gi...@apache.org>.
ozlerhakan commented on issue #618:
URL: https://github.com/apache/solr-operator/issues/618#issuecomment-1778947251
Sorry that I totally missed your message. I'm unsure whether this could be applicable from the implementation perspective. Would it be possible to introduce another handy CRD for custom Solr settings that interacts with the SolrCloud CRD at pod startup?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [I] Role Assignment for SOLR Replicas [solr-operator]
Posted by "ozlerhakan (via GitHub)" <gi...@apache.org>.
ozlerhakan commented on issue #618:
URL: https://github.com/apache/solr-operator/issues/618#issuecomment-1748346189
When do you think you can consider this enhancement to be part of one of the next releases @HoustonPutman ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [I] Role Assignment for SOLR Replicas [solr-operator]
Posted by "msobkowiak-olx (via GitHub)" <gi...@apache.org>.
msobkowiak-olx commented on issue #618:
URL: https://github.com/apache/solr-operator/issues/618#issuecomment-2661119498
Hey there ! Wanted to ask if technically this can work for this problem
- introduce additional fields to the spec:
- number of data nodes
- number of coordinator nodes
- number of overseer nodes
- run those as additional statefulsets (different pod naming) sharing the same configiuration except for additional parameter -Dsolr.node.roles=<insert configuration for pod type>, eg for overseer we would have as a result:
- solr-overseer-0 (-Dsolr.node.roles=data:off,overseer:allowed,coordinator:off)
...
- solr-overseer-n
- solr-coordinator-0 (-Dsolr.node.roles=data:off,overseer:disallowed,coordinator:on)
...
- solr-coordinator-n
- solr-data-0 (-Dsolr.node.roles=data:on,overseer:disallowed,coordinator:off)
...
- solr-data-n
This would be of course breaking change, it could also be a separated CRD (eg. SolrCloudWithRoles or so)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org