You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@accumulo.apache.org by "keith-turner (via GitHub)" <gi...@apache.org> on 2023/02/01 00:23:09 UTC
[GitHub] [accumulo] keith-turner opened a new issue, #3178: Support balancing tablets using named tserver groups
keith-turner opened a new issue, #3178:
URL: https://github.com/apache/accumulo/issues/3178
Accumulo currently supports allocating tablets to specific servers using IP addresses and hostnames using certain balancer plugins. This provides a way to dedicate compute resources to specific tables. Another way this could be accomplished is by adding a group option to tablet servers like [scan servers](https://github.com/apache/accumulo/blob/ab8e2f518887a2505cad9dbc02e82197babfc1bd/server/tserver/src/main/java/org/apache/accumulo/tserver/ScanServer.java#L130) have. The group set for a tablet server could then be made available to balancer plugins. Balancer plugins could use this information to assign specific tables or tablets to groups of tablet servers.
This feature would be useful for running groups of tablet servers in Kubernetes pods where the IP addresses may not be static. Then tables could be assigned to those different groups giving different tables different qualities of compute resources. It could also be used to simplify the balancer configuration on clusters with fixed ip addresses.
An example of of this could work is
* on host A run `accumulo tserver -g meta`
* on host B run `accumulo tserver -g meta`
* on host C run `accumulo tserver`
* on host D run `accumulo tserver`
After doing the above configure a balancer to assign the accumulo metadata table to group meta. This would result in accumulo metadata tables going to host A or B without having to configure the balancer with specifig information about host A and B. The balancer plugin should assign all other tablets to host C and D.
To implement this feature would need to :
- Add the `-g` startup option to the tserver
- Store the group name in zookeeper for each tserver
- Modify the manager to get the group names from zookeeper and pass them to the balancer
- Possibly update the accumulo-cluster script to support group names for tservers in the cluster yaml file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] ctubbsii commented on issue #3178: Support balancing tablets using named tserver groups
Posted by "ctubbsii (via GitHub)" <gi...@apache.org>.
ctubbsii commented on issue #3178:
URL: https://github.com/apache/accumulo/issues/3178#issuecomment-1421367985
In any case, after looking at #3189, it doesn't look like it will add too much complexity to add a service group name for all services. So, I think my initial reservations, based on my concert about complexity, aren't really valid. I still think it's useful to think in terms of the complexity / convenience trade-offs, but after seeing the implementation, my opinion has swung the other way in terms of where I think this would fall.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] ddanielr commented on issue #3178: Support balancing tablets using named tserver groups
Posted by "ddanielr (via GitHub)" <gi...@apache.org>.
ddanielr commented on issue #3178:
URL: https://github.com/apache/accumulo/issues/3178#issuecomment-1421390468
> > > Right, but couldn't these groups just be kubernetes metadata? Isn't that already something that kubernetes supports today?
> >
> >
> > I don't know. Looking at [this](https://stackoverflow.com/questions/63169737/kubernetes-api-list-pods-with-a-label) you can query the Kubernetes API server, but I don't know all of the intricacies involved with what appears to be auth tokens, the type of results that are returned, latency, etc.
>
> You can use the [downward api](https://kubernetes.io/docs/concepts/workloads/pods/downward-api/) to expose some pod info (including labels and annotations) via files or environment variables inside the pod. I'm wondering if that would be sufficient?
You can do this pretty easily in your pod definition with setting a metadata label.
The common approach is having the balancer process hit the k8s API using a service account to query for all pods with a specific label type.
However, If you want to reduce the balancer's "knowledge" of k8s existing then you could inject the label metadata via a environment variable in your pod spec set to the same label value.
This would still require the balancer to understand the concept of groups, but remove the k8s requirement from accumulo-based code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] ctubbsii commented on issue #3178: Support balancing tablets using named tserver groups
Posted by "ctubbsii (via GitHub)" <gi...@apache.org>.
ctubbsii commented on issue #3178:
URL: https://github.com/apache/accumulo/issues/3178#issuecomment-1421300022
@keith-turner wrote:
> In the kubernetes case this is not so easy when you have an auto-scaled group of tservers that is dynamically allocated ip addresses. When running scan servers and compactors in kubernetes it was so easy to start them with a group specified on the command line and have the rest of the system respond automatically as those different groups scaled up and down.
It seems like it would be easy to make a balancer that could detect some tag/label associated with a particular kubernetes group. I'm not that familiar with running stuff in kubernetes, but I've seen the concept of "genders" or "flavors" or "groups" in scalable architectures before, so I'd be surprised if kubernetes didn't support something like that that the user could leverage easily on their own.
@dlmarion wrote:
> While the HostRegexTableLoadBalancer might work in an environment like kubernetes, I don't think it's user friendly.
I agree. I don't necessarily think that balancer should be used. I think a different balancer that could make use of different labels would be warrented. However, given this is a niche use case, I still think this would be better suited to having a custom balancer for that situation... which is why we have balancers pluggable... rather than architecting a custom solution inside Accumulo. I don't think we need to be responsible for all user conveniences... making pluggable endpoints that empower users is often sufficient; there are trade-offs that grow our code complexity and increase our maintenance burden the more we try to take on every possible user deployment use case. If it's going to increase our code complexity, I'm going to favor more towards empowering users to customize things using our pluggable endpoints, rather than adding more APIs and config to Accumulo. I think it's worth considering keeping this a pluggable solution rather than a built-in one.
> If we have the ability to assign group labels to TabletServers (like we do for ScanServers, and "queues" for Compactors), then for a kubernetes deployment the user has to:
>
> 1. Create different yaml descriptor files for the Accumulo processes to assign different groups
> 2. Create a load balancer that balances tables using the groups (a variation of the HostRegexTableLoadBalancer)
> 3. Configure this new balancer, assigning groups to tables.
Right, but couldn't these groups just be kubernetes metadata? Isn't that already something that kubernetes supports today? Do we need to add baked-in metadata for tservers to define tserver groups or can users already get that today with some kind of kubernetes labels? If they already label tservers today in kubernetes, then growing the complexity of Accumulo to achieve this is much less appealing to me. But, I would be in favor of writing a blog post to show off a balancer implementation that would be aware of an existing kubernetes label feature. I think that would be very useful to empower users by showing them what our pluggable endpoints are capable of.
> I have been working on a solution for this, I should be able to put it up today for comments.
I will take a look at #3189.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] brianloss commented on issue #3178: Support balancing tablets using named tserver groups
Posted by "brianloss (via GitHub)" <gi...@apache.org>.
brianloss commented on issue #3178:
URL: https://github.com/apache/accumulo/issues/3178#issuecomment-1421376055
> > Right, but couldn't these groups just be kubernetes metadata? Isn't that already something that kubernetes supports today?
>
> I don't know. Looking at [this](https://stackoverflow.com/questions/63169737/kubernetes-api-list-pods-with-a-label) you can query the Kubernetes API server, but I don't know all of the intricacies involved with what appears to be auth tokens, the type of results that are returned, latency, etc.
You can use the [downward api](https://kubernetes.io/docs/concepts/workloads/pods/downward-api/) to expose some pod info (including labels and annotations) via files or environment variables inside the pod. I'm wondering if that would be sufficient?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] dlmarion closed issue #3178: Support balancing tablets using named tserver groups
Posted by "dlmarion (via GitHub)" <gi...@apache.org>.
dlmarion closed issue #3178: Support balancing tablets using named tserver groups
URL: https://github.com/apache/accumulo/issues/3178
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] dlmarion commented on issue #3178: Support balancing tablets using named tserver groups
Posted by "dlmarion (via GitHub)" <gi...@apache.org>.
dlmarion commented on issue #3178:
URL: https://github.com/apache/accumulo/issues/3178#issuecomment-1420654933
> I think that's something users can already do outside of Accumulo (such as with the hostname, or some other external group file) and using either the HostRegexTableLoadBalancer or a custom balancer.
While the HostRegexTableLoadBalancer might work in an environment like kubernetes, I don't think it's user friendly. You can pass the kubernetes pod ip address to the Accumulo server process using the `-a` argument, but the ip address for the pod is based on the the range of ip addresses assigned to the host. When deploying Accumulo in this environment and using the HostRegexTableLoadBalancer the user would need to:
1. Create different yaml descriptor files for the Accumulo processes to assign different groups to different nodes using kubernetes `nodeSelector` type in the yaml file.
2. Find the IP range for all of the nodes involved
3. For each table that they want to configure, create a regex using the IP ranges of the hosts involved.
If we have the ability to assign group labels to TabletServers (like we do for ScanServers, and "queues" for Compactors), then for a kubernetes deployment the user has to:
1. Create different yaml descriptor files for the Accumulo processes to assign different groups
2. Create a load balancer that balances tables using the groups (a variation of the HostRegexTableLoadBalancer)
3. Configure this new balancer, assigning groups to tables.
Then, the kubernetes administrator can scale the deployments of a particular group as needed without worrying about the node's IP address. In a commercial cloud environment, I can imagine that node IP addresses may change over time which could be really burdensome.
I have been working on a solution for this, I should be able to put it up today for comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] keith-turner commented on issue #3178: Support balancing tablets using named tserver groups
Posted by "keith-turner (via GitHub)" <gi...@apache.org>.
keith-turner commented on issue #3178:
URL: https://github.com/apache/accumulo/issues/3178#issuecomment-1420092874
> I think that's something users can already do outside of Accumulo (such as with the hostname, or some other external group file) and using either the HostRegexTableLoadBalancer or a custom balancer.
In the kubernetes case this is not so easy when you have an auto-scaled group of tservers that is dynamically allocated ip addresses. When running scan servers and compactors in kubernetes it was so easy to start them with a group specified on the command line and have the rest of the system respond automatically as those different groups scaled up and down.
Even for clusters of machines I think this group option is nice because it makes the Accumulo config indifferent to the specifics of the cluster. So I can use the same accumulo config on two different cluster with different IP address+hostnames and if I start 10 tservers in group A and 20 in group B on each cluster, then same config will behave the same on both clusters. Where with the regex config I have to adapt the config to the cluster or try to adapt the clusters ips and hostnames to the config. So it makes it easier to have reproducible behavior across clusters.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] ctubbsii commented on issue #3178: Support balancing tablets using named tserver groups
Posted by "ctubbsii (via GitHub)" <gi...@apache.org>.
ctubbsii commented on issue #3178:
URL: https://github.com/apache/accumulo/issues/3178#issuecomment-1421399146
> However, If you want to reduce the balancer's "knowledge" of k8s existing then you could inject the label metadata via a environment variable in your pod spec set to the same label value.
That's basically the direction #3189 takes... adding environment, but not a process environment variable, but environment in Accumulo's service advertisements... which is a lot easier for remote services to see than environment inside the server.
>
> This would still require the balancer to understand the concept of groups, but remove the k8s requirement from accumulo-based code.
Yeah, a group-aware balancer has yet to be written, but #3189 lays the groundwork in the environment so it can be done subsequently, all unaware of k8.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] dlmarion commented on issue #3178: Support balancing tablets using named tserver groups
Posted by "dlmarion (via GitHub)" <gi...@apache.org>.
dlmarion commented on issue #3178:
URL: https://github.com/apache/accumulo/issues/3178#issuecomment-1414117540
The [HostRegexTableLoadBalancer](https://github.com/apache/accumulo/blob/main/core/src/main/java/org/apache/accumulo/core/spi/balancer/HostRegexTableLoadBalancer.java) allows the user to define groups of tablet servers (by hostname of ip regex) across which to balance a table's tablets. The sets could be overlapping, or not, and all hosts *not* covered by a regex are part of a default group that is used to balance unconfigured table's tablets.
If we support setting multiple groups via the `-g` argument, then the `HostRegexTableLoadBalancer` could be deprecated in favor of the new balancer thas has the same functionality.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] ddanielr commented on issue #3178: Support balancing tablets using named tserver groups
Posted by "ddanielr (via GitHub)" <gi...@apache.org>.
ddanielr commented on issue #3178:
URL: https://github.com/apache/accumulo/issues/3178#issuecomment-1421488470
I looked into the HostRegexLoadBalancer and created an issue that would help improve the user-friendliness.
I think it's much smaller in scope to this proposed change so I thought it might be a benefit to 2.1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] dlmarion commented on issue #3178: Support balancing tablets using named tserver groups
Posted by "dlmarion (via GitHub)" <gi...@apache.org>.
dlmarion commented on issue #3178:
URL: https://github.com/apache/accumulo/issues/3178#issuecomment-1421324707
> Right, but couldn't these groups just be kubernetes metadata? Isn't that already something that kubernetes supports today?
I don't know. Looking at [this](https://stackoverflow.com/questions/63169737/kubernetes-api-list-pods-with-a-label) you can query the Kubernetes API server, but I don't know all of the intricacies involved with what appears to be auth tokens the type of results are returned, latency, etc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] keith-turner commented on issue #3178: Support balancing tablets using named tserver groups
Posted by "keith-turner (via GitHub)" <gi...@apache.org>.
keith-turner commented on issue #3178:
URL: https://github.com/apache/accumulo/issues/3178#issuecomment-1414710310
> The sets could be overlapping, or not, and all hosts not covered by a regex are part of a default group that is used to balance unconfigured table's tablets.
What are the use cases for overlapping groups of servers?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] dlmarion commented on issue #3178: Support balancing tablets using named tserver groups
Posted by "dlmarion (via GitHub)" <gi...@apache.org>.
dlmarion commented on issue #3178:
URL: https://github.com/apache/accumulo/issues/3178#issuecomment-1415789072
> What are the use cases for overlapping groups of servers?
I don't remember, if I had to guess it was just maximum flexibility. Thinking about this further, we might not need to support multiple groups per server. When configuring the HostRegexTableLoadBalancer (see [test](https://github.com/apache/accumulo/blob/main/core/src/test/java/org/apache/accumulo/core/spi/balancer/HostRegexTableLoadBalancerTest.java#L205)), the user maps a table directly to a regex that maps to hosts. Assigning a group name to a TabletServer provides a level of indirection. For example:
* There could be a set of TabletServers that use the group `meta` to host the metadata table
* There could be a set of TabletServers that use the group `appA` to host the tables used by application A
* There could be a set of TabletServers that use the group `appsBC` to host the tables used by applications B & C
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] ctubbsii commented on issue #3178: Support balancing tablets using named tserver groups
Posted by "ctubbsii (via GitHub)" <gi...@apache.org>.
ctubbsii commented on issue #3178:
URL: https://github.com/apache/accumulo/issues/3178#issuecomment-1420037129
It seems to me that the core characteristic of this idea is that tservers themselves are annotated with some kind of label/group identifier. I think that's something users can already do outside of Accumulo (such as with the hostname, or some other external group file) and using either the HostRegexTableLoadBalancer or a custom balancer.
I don't think we need to make any architectural or built-in changes to Accumulo to support this use case, and I think if we try to, we're just going to add unnecessary complexity to maintain, with the end result being no more functionality than we already have that is maybe only *slightly* easier to use for some people. I'm really not sure that it's worth introducing more complex code for that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] dlmarion commented on issue #3178: Support balancing tablets using named tserver groups
Posted by "dlmarion (via GitHub)" <gi...@apache.org>.
dlmarion commented on issue #3178:
URL: https://github.com/apache/accumulo/issues/3178#issuecomment-1421497095
be sure to look at the [GroupBalancer](https://github.com/apache/accumulo/blob/main/core/src/main/java/org/apache/accumulo/core/spi/balancer/GroupBalancer.java), it may be applicable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org