You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@accumulo.apache.org by "dlmarion (via GitHub)" <gi...@apache.org> on 2024/02/06 17:07:50 UTC
[I] Investigate moving log sort functionality out of tablet servers. [accumulo]
dlmarion opened a new issue, #4232:
URL: https://github.com/apache/accumulo/issues/4232
I just had a discussion with @ddanielr about #3274. We went on a tangent and ended up taking about how logs are sorted by random tservers. Dan brought up that for the elasticity changes we may want to consider moving log sorting out of tservers into its own process that could be autoscaled and isolated from immediate scans.
Update : an alternative to moving log sort to its own process would be to make it resource group aware. So make log sort happen within a specific resource group that makes sense for the particular WAL. Currently when sorting a log the entire set of tablet servers may be used. Ideally if tablet server A in resource group RG1 dies, the later when its logs are sorted it would be nice if that was done by tservers in RG1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Investigate moving log sort functionality out of tablet servers. [accumulo]
Posted by "dlmarion (via GitHub)" <gi...@apache.org>.
dlmarion commented on issue #4232:
URL: https://github.com/apache/accumulo/issues/4232#issuecomment-1932111156
Moving the resource group into the path will make the client slower as it will cause more back and forth ZK operations when trying to get a list of all of the tservers or sserver. For example [here](https://github.com/apache/accumulo/blob/elasticity/core/src/main/java/org/apache/accumulo/core/clientImpl/ClientContext.java#L405) and [here](https://github.com/apache/accumulo/blob/elasticity/core/src/main/java/org/apache/accumulo/core/rpc/clients/TServerClient.java#L57)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Investigate moving log sort functionality out of tablet servers. [accumulo]
Posted by "ddanielr (via GitHub)" <gi...@apache.org>.
ddanielr commented on issue #4232:
URL: https://github.com/apache/accumulo/issues/4232#issuecomment-1932047892
Would it make sense to move the resource group up in the path vs having it as an attribute on the specific server type?
i.e.
`ZKROOT / <resource group> / compactors / cp1`
`ZKROOT / <resource group> / sservers / ss1`
`ZKROOT / <resource group> / tservers / ts1`
Vs
`ZKROOT / compactors / <resource group> / cp1 `
`ZKROOT / sservers / ss1 / <resource group> `
`ZKROOT / tservers / <resource group> / ts1`
I'm not that familiar with structuring items inside of zookeeper. However, I could see this being a benefit if we wanted to have any other `resource group` specific things (properties, service discovery, etc) stored in ZK.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Investigate moving log sort functionality out of tablet servers. [accumulo]
Posted by "dlmarion (via GitHub)" <gi...@apache.org>.
dlmarion commented on issue #4232:
URL: https://github.com/apache/accumulo/issues/4232#issuecomment-1943736188
Closed with the merge of #4239. We can create a new issue for making the log sorting processes resource group aware at a later time if that makes sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Investigate moving log sort functionality out of tablet servers. [accumulo]
Posted by "keith-turner (via GitHub)" <gi...@apache.org>.
keith-turner commented on issue #4232:
URL: https://github.com/apache/accumulo/issues/4232#issuecomment-1932150333
> Looking at references to Constants.TSERVERS, we would have to start passing around the the resource group name along with the tserver HostAndPort or address
I am uncertain if that is the best way to make log recovery RG aware. Another way would be when a tablet is seen w/ walogs to ask the balancer which tablet server to assign to and use the RG of that tserver for log sort (but not actually use the result from the balancer for assignment). There are pros and cons to each method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Investigate moving log sort functionality out of tablet servers. [accumulo]
Posted by "keith-turner (via GitHub)" <gi...@apache.org>.
keith-turner commented on issue #4232:
URL: https://github.com/apache/accumulo/issues/4232#issuecomment-1932116176
> I think to do this efficiently we would need to modify the way that TabletServers and ScanServers are registered in ZooKeeper.
Making log recovery RG aware could be a big refactor. I think doing that refactor is a lower priority than the current work that is in progress to make sure log recovery happens somewhere.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Investigate moving log sort functionality out of tablet servers. [accumulo]
Posted by "dlmarion (via GitHub)" <gi...@apache.org>.
dlmarion commented on issue #4232:
URL: https://github.com/apache/accumulo/issues/4232#issuecomment-1931969315
```
1. When a walog is created include the tablet server and its current RG in its name. Currently the name only includes the tserver
2. If a log ever needs recovery perfer tablet servers in the same RG
3. If there are no tservers present in the RG, then use any server (tsever,compactor, scan server)
```
I think to do this efficiently we would need to modify the way that TabletServers and ScanServers are registered in ZooKeeper. The [Compactor](https://github.com/apache/accumulo/blob/elasticity/server/compactor/src/main/java/org/apache/accumulo/compactor/Compactor.java#L250) ZooKeeper path has the resource group in it, but the [TabletServer](https://github.com/apache/accumulo/blob/elasticity/server/tserver/src/main/java/org/apache/accumulo/tserver/TabletServer.java#L494) and [ScanServer](https://github.com/apache/accumulo/blob/elasticity/server/tserver/src/main/java/org/apache/accumulo/tserver/ScanServer.java#L305) do not. I'm not sure of the consequences of making this change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Investigate moving log sort functionality out of tablet servers. [accumulo]
Posted by "dlmarion (via GitHub)" <gi...@apache.org>.
dlmarion closed issue #4232: Investigate moving log sort functionality out of tablet servers.
URL: https://github.com/apache/accumulo/issues/4232
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Investigate moving log sort functionality out of tablet servers. [accumulo]
Posted by "dlmarion (via GitHub)" <gi...@apache.org>.
dlmarion commented on issue #4232:
URL: https://github.com/apache/accumulo/issues/4232#issuecomment-1932121042
> Making log recovery RG aware could be a big refactor. I think doing that refactor is a lower priority than the current work that is in progress to make sure log recovery happens somewhere.
I agree. Looking at references to Constants.TSERVERS, we would have to start passing around the the resource group name along with the tserver HostAndPort or address String in a lot of methods.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Investigate moving log sort functionality out of tablet servers. [accumulo]
Posted by "dlmarion (via GitHub)" <gi...@apache.org>.
dlmarion commented on issue #4232:
URL: https://github.com/apache/accumulo/issues/4232#issuecomment-1930431271
> an alternative to moving log sort to its own process would be to make it resource group aware.
Processes currently know their resource group, but group name for tables is not a first-class citizen. There is a custom property in TableLoadBalancer (`table.custom.assignment.group`) that the balancer uses. Nothing else uses this as far as I know.
>So make log sort happen within a specific resource group that makes sense for the particular WAL.
If we made the property a first-class citizen of the table configuration, then I think there are cases where recovery may never happen. Consider the case of a user deciding to change the way they access a table from IMMEDIATE to EVENTUAL. They shut down all the tablet servers and decide to bulk load and use scan servers. Some of the TabletServers don't stop cleanly so those tablets have walogs that may not get sorted and recovered.
I have been thinking of a different approach because I think there is a danger of recovery not occurring. We made changes in #4229 to host a tablet regardless of availability for the purposes of recovery. We did this because if recovery does not happen, then ScanServers will not return the data in the walogs, the Compactor won't compact the data in the walogs, and the GC process will never delete the walog files. My proposal would be to embed the LogSorter in the Compactor and ScanServer in addition to the it's current location in the TabletServer. I think the Compactor processes should look for log sorting work before it starts a new compaction, and I think we setup the ScanServer to use the LogSorter just like the TabletServer does. The idea being that resource groups allow the user to dedicate resources to tables for specific things, but when a failure happens and recovery needs to occur all members participate for the accuracy of the data and safety to the system.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Investigate moving log sort functionality out of tablet servers. [accumulo]
Posted by "keith-turner (via GitHub)" <gi...@apache.org>.
keith-turner commented on issue #4232:
URL: https://github.com/apache/accumulo/issues/4232#issuecomment-1930478684
> My proposal would be to embed the LogSorter in the Compactor and ScanServer in addition to the it's current location in the TabletServer.
One thing to consider when doing that would be memory usage as compactors may be configured with less memory than tablet servers. I don't think memory would be a concern because log sort pulls chunks of data into memory and sorts them possibly creating multiple sorted files for a single unsorted log. However that assumption is based on how the code used to work, have not looked at it recently. Would be good to examine the current memory usage of log sort.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Investigate moving log sort functionality out of tablet servers. [accumulo]
Posted by "keith-turner (via GitHub)" <gi...@apache.org>.
keith-turner commented on issue #4232:
URL: https://github.com/apache/accumulo/issues/4232#issuecomment-1930909645
> If we made the property a first-class citizen of the table configuration, then I think there are cases where recovery may never happen.
Do not necessarily need to make it a first class citizen of the table to make log recovery RG(resource group) aware. Could do something like the followng.
1. When a walog is created include the tablet server and its current RG in its name. Currently the name only includes the tserver
2. If a log ever needs recovery perfer tablet servers in the same RG
3. If there are no tservers present in the RG, then use any server (tsever,compactor, scan server)
This makes log recovery use the same RG as where it was created under normal circumstances and falls back to other resources under unusual circumstances.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org