[GitHub] [accumulo] ctubbsii commented on issue #2187: Sorted Log Recovery configuration design

[GitBox <gi...@apache.org>]

ctubbsii commented on issue #2187:
URL: https://github.com/apache/accumulo/issues/2187#issuecomment-875791496


   Our config tends to be a big massive pool of properties. A challenge is to figure out how to "localize" subsets of the config to the specific scope of the components that utilize it. The way we've done this is by heavily relying on the hierarchical nature of our properties, and using common prefixes to store configuration properties that are related. However, these prefixes can be stripped off when the properties are "localized" to the code that actually reads the properties' values to perform its function.
   
   We currently localize some properties by using `AccumuloConfiguration.getAllPropertiesWithPrefixStripped(Property contextSpecificPrefix)` for extracting and localizing scan dispatch configuration and some compaction configuration.
   
   The fact that we can "localize" sets of properties this way pretty easily, means we can also have the same set of properties stored in different contexts, established with different prefixes. For example, `table.file.*` and `tserver.wal.sort.file.*`.
   
   So, all we need to do is figure out the set of properties that RFile read/write code needs locally, and then make sure we can extract them from the big pool using a common prefix for different contexts, as in:
   
   ```java
   var tableFileConf = FileConfig.fromMap(tableConf.getAllPropertiesWithPrefixStripped("table.file."));
   var walFileConf = FileConfig.fromMap(systemConf.getAllPropertiesWithPrefixStripped("tserver.wal.sort.file."));
   ```
   
   This requires thinking about config that is "local" to RFile operations, and perhaps creating a concrete RFileConfiguration type for use in its public API and other related APIs (like AccumuloFileOutputFormat).
   
   Alternatively, the simplest thing to do is do nothing, and just read the per-table settings from the system config, as though these RFiles are logically associated with a generic, system-wide, non-existent table, rather than a specific sorted WAL context. We certainly have existing code that behaves this way, but it's less than ideal, and can create confusion for users.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org