You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@accumulo.apache.org by "ddanielr (via GitHub)" <gi...@apache.org> on 2023/12/06 22:26:05 UTC
[I] Modify the DefaultCompactionPlanner to set priorities at a table name level [accumulo]
ddanielr opened a new issue, #4033:
URL: https://github.com/apache/accumulo/issues/4033
We currently create three compaction services for `root`, `meta`, and a `default` compaction service for a running instance of accumulo.
If the DefaultCompactionPlanner is modified to use table names in addition to file sizes, then a single compaction service could be used instead of creating three different compaction services.
Priority levels would be something like `root:1, meta:2, accumulo*:3, user:4` where 1 is the highest priority.
This would allow a single compaction service to be used when starting up a instance of accumulo.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Modify the DefaultCompactionPlanner to set priorities at a table name level [accumulo]
Posted by "ddanielr (via GitHub)" <gi...@apache.org>.
ddanielr closed issue #4033: Modify the DefaultCompactionPlanner to set priorities at a table name level
URL: https://github.com/apache/accumulo/issues/4033
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Modify the DefaultCompactionPlanner to set priorities at a table name level [accumulo]
Posted by "keith-turner (via GitHub)" <gi...@apache.org>.
keith-turner commented on issue #4033:
URL: https://github.com/apache/accumulo/issues/4033#issuecomment-1845642261
> Is there a reason to limit the size to 16 bits?
I think it was mainly driven by this [map](https://github.com/apache/accumulo/blob/3e26064bfde82986798d45c0aa4adc9ca82c476b/server/compaction-coordinator/src/main/java/org/apache/accumulo/coordinator/QueueAndPriority.java#L27). We used to have a long for the priority, but @dlmarion and I realized that if the priority as a long was extremely high cardinality that could cause an OOME. We could not think of a good reason to have an extremely high cardinality priority so switched the long to a short.
> we could find in the future that an accumulo.hosting or something outside of the metadata that provides explicit elasticity information. Things that may want a finer level of control than being lumped into accumulo.*.
The computation of the priority is done by a pluggable component, the CompactionPlanner. So it other situations could be handled, they could compute the priority in anyway they like.
> Something like byte for table type, byte for compaction type, and 2 bytes for count would fit into 32 bits and allow for growth. Limits for file counts could be limited to use less that the full 2^16 if necessary.
I think many situations like this can be handled with the short. May need to do things like take the log or sqrt of a value you are putting in the short to compress the information while still maintaining sort order. The priority does not need to be super precise, just getting things in the neighborhood is usually good enough.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Modify the DefaultCompactionPlanner to set priorities at a table name level [accumulo]
Posted by "keith-turner (via GitHub)" <gi...@apache.org>.
keith-turner commented on issue #4033:
URL: https://github.com/apache/accumulo/issues/4033#issuecomment-1845558890
To implement this will need to modify [CompactionJobPrioritizer.java](https://github.com/apache/accumulo/blob/3e26064bfde82986798d45c0aa4adc9ca82c476b/core/src/main/java/org/apache/accumulo/core/util/compaction/CompactionJobPrioritizer.java#L32). Currently this class creates a 16 bit priority using the first bit for compaction type and the remaining 15 bits for the number of files. Could change it to use the 16 bits in the following way.
* First 2 bits are table type, accumulo.root, accumulo.metadata, accumulo.*, user table
* Third bit is compaction type
* Remaining 13 bits are the files count. Would need to adjust the max checks to 2^13.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Modify the DefaultCompactionPlanner to set priorities at a table name level [accumulo]
Posted by "EdColeman (via GitHub)" <gi...@apache.org>.
EdColeman commented on issue #4033:
URL: https://github.com/apache/accumulo/issues/4033#issuecomment-1845601599
Is there a reason to limit the size to 16 bits? Thinking that we might want more room than 2-bits for table type to allow for expansion - one example would be to include accumulo.fate (or whatever it gets called) and we could find in the future that an accumulo.hosting or something outside of the metadata that provides explicit elasticity information. Things that may want a finer level of control than being lumped into accumulo.*.
Something like byte for table type, byte for compaction type, and 2 bytes for count would fit into 32 bits and allow for growth. Limits for file counts could be limited to use less that the full 2^16 if necessary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Modify the DefaultCompactionPlanner to set priorities at a table name level [accumulo]
Posted by "ddanielr (via GitHub)" <gi...@apache.org>.
ddanielr commented on issue #4033:
URL: https://github.com/apache/accumulo/issues/4033#issuecomment-1881835035
Repointed this to elasticity because 3.1 still has the concept of internal compactions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org