You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@accumulo.apache.org by "ddanielr (via GitHub)" <gi...@apache.org> on 2026/07/16 12:36:38 UTC
[I] Switch sort order for metadata table entries [accumulo]
ddanielr opened a new issue, #6479:
URL: https://github.com/apache/accumulo/issues/6479
Currently, when a table is created it uses a one-up counter for its table name. This means that a workload that creates new tables end up putting all that work on the same metadata tablet.
**Describe the solution you'd like**
Switch the sort order of the metadata table to use the least significant bit of the tableID.
This would spread the new tables metadata entries across multiple metadata tablets while still allowing each table section to be ordered.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Switch sort order for metadata table entries [accumulo]
Posted by "ctubbsii (via GitHub)" <gi...@apache.org>.
ctubbsii commented on issue #6479:
URL: https://github.com/apache/accumulo/issues/6479#issuecomment-5209324129
@keith-turner suggested:
> could start doing `<random base 36 char><existing one up table id>` for new table IDs. This would avoid collision with existing table IDs and spread new tables. Would not spread existing tables though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Switch sort order for metadata table entries [accumulo]
Posted by "dlmarion (via GitHub)" <gi...@apache.org>.
dlmarion commented on issue #6479:
URL: https://github.com/apache/accumulo/issues/6479#issuecomment-5252862172
I wonder if we can just change the starting table id number stored in ZooKeeper on upgrade and then reverse the characters of the resulting string in `Utils.getNextId` like @ctubbsii mentioned. I think this is a similar approach described in the issue description, but by using a higher number we can deal with characters and not bits.
If we assume pre-upgrade that users have split their metadata table sufficiently, then we don't need to consider pre-existing tables. We would not need to do anything on upgrade to the metadata table, just change the starting number in ZooKeeper to `100` or `1000` if it's not already that high. We would not need to add a prefix or suffix character to ensure unique table ids either.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Switch sort order for metadata table entries [accumulo]
Posted by "dlmarion (via GitHub)" <gi...@apache.org>.
dlmarion commented on issue #6479:
URL: https://github.com/apache/accumulo/issues/6479#issuecomment-5026902886
Would this mean that we need to rewrite the metadata table on the upgrade to 4.0? If the issue is the one-up number creating hot spots in the metadata table, then let's change that instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Switch sort order for metadata table entries [accumulo]
Posted by "ctubbsii (via GitHub)" <gi...@apache.org>.
ctubbsii commented on issue #6479:
URL: https://github.com/apache/accumulo/issues/6479#issuecomment-5148829390
I think this may have come from a passing comment I made to @ddanielr a few weeks ago. If that's the case, then the idea I had in mind was to simply serialize the table IDs in the metadata table in reverse character order, so that they naturally would spread out more (table 123 would be next to table 43 and c3, but table 124 would be next to a4 and 94). That way, two tables created in succession would be less likely to share the same metadata tablets, but all the table's metadata would still be adjacent for any given table.
Implementing this would require rewriting metadata on upgrade, or storing some extra information in ZK and the metadata table so we knew which format a particular table's metadata entries were using. Both of those seemed undesirable, so I kind of dismissed the idea shortly after I said it.
I'm also not really sure that this would really improve things much, because if the number of tables is sufficiently dense and table "busy-ness" is about the same (or at least, hard to predict), placing one table's metadata next to an older table's metadata may not actually be better than placing it next to the table that was most recently created. And that's another reason I kind of dismissed the idea.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Switch sort order for metadata table entries [accumulo]
Posted by "phrocker (via GitHub)" <gi...@apache.org>.
phrocker commented on issue #6479:
URL: https://github.com/apache/accumulo/issues/6479#issuecomment-4997028457
Is table creation the load or the fate ops / zk round trips, lock acquisition, fsyncs or is the pressure downstream at metadata splits? if scattering is the goal could you get the same goal by doing it at id assignment instead of in the metadata sort order?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Switch sort order for metadata table entries [accumulo]
Posted by "ctubbsii (via GitHub)" <gi...@apache.org>.
ctubbsii commented on issue #6479:
URL: https://github.com/apache/accumulo/issues/6479#issuecomment-5209410999
Since we're thinking about adding a character, it would be pretty easy to guarantee no collisions if we just start or end every new TableId with an underscore and reverse the other base36 digits from the 1-up sequence. Other characters might cause problems with HDFS paths. Could also use `t_` as a prefix if 2 characters are okay, and `n_` for new namespace IDs.
My previous reservation was about translating between the stored bytes and the "actual" table ID (table ID "abc" stored as "cba"), but if we're not translating, and only deciding the table ID at creation time, then this concern goes away (table ID "t_cba" is simply the next table ID chosen after "t_bba", and it is the actual table ID, no need to translate).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Switch sort order for metadata table entries [accumulo]
Posted by "ctubbsii (via GitHub)" <gi...@apache.org>.
ctubbsii commented on issue #6479:
URL: https://github.com/apache/accumulo/issues/6479#issuecomment-5287285171
> I think this is a similar approach described in the issue description, but by using a higher number we can deal with characters and not bits.
My intent was always to reverse the base-36 characters, not the bits. 😺
> If we assume pre-upgrade that users have split their metadata table sufficiently, then we don't need to consider pre-existing tables.
I'm okay with that assumption.
> We would not need to do anything on upgrade to the metadata table, just change the starting number in ZooKeeper to `100` or `1000` if it's not already that high. We would not need to add a prefix or suffix character to ensure unique table ids either.
Do you mean simply jump to the next order of magnitude, so that the number of digits in the sequence number is always greater than any previously assigned ID? That could work, but without a prefix, we need to be careful that we don't incorrectly parse something with leading zeros. That was never possible with previous IDs. I think if we're treating the IDs as a canonical byte string, it's fine, but if there's any code that is relying on it being a base-36 number (other than the function to increment the counter before we reverse it), then we need to be careful. This is something that we'd have had to make sure even if we did have a prefix, but without a prefix, it's less obvious.
This would allow new tables to be nicely interspersed with the existing tables in the metadata, except that we'd have a bunch of new metadata starting with zeroes at the start of the metadata table. That's probably okay because it should split at some point.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org