[GitHub] [accumulo] ctubbsii opened a new issue, #3359: Drop Mapfile support

["ctubbsii (via GitHub)" <gi...@apache.org>]

ctubbsii opened a new issue, #3359:
URL: https://github.com/apache/accumulo/issues/3359

   This task is to drop support for Mapfiles, and to enforce the use of RFiles for storing Accumulo table data. Map files haven't been the default in a long time, and I'm not even really sure that they work, as Accumulo testing pretty exclusively uses RFiles.
   
   We should refuse to upgrade to Accumulo 3.0 if any of these preconditions fail to be met:
   
   1. `table.file.type` is set to anything other than `RFile.EXTENSION (rf)` (could refuse to start Accumulo by adding a configuration validator to force the use of RFiles)
   2. Any files referenced in the `accumulo.root` or `accumulo.metadata` table have an extension other than `RFile.EXTENSION (rf)` (or a related tmp file)
   
   This would enable us to remove the code that supports Map files.
   
   Users could receive a message when trying to upgrade that they cannot upgrade, but must instead first compact the tablets containing Map files (we could list the table names) after ensuring `table.file.type` is set to `RFile.EXTENSION (rf)`
   
   Reasons to remove Map file support:
   
   1. Uncertain current support state (no idea if it even works)
   2. Poor support for features available in RFiles like column family filtering, multi-level indexing, sampling, etc.,
   3. Doesn't work with RFile encryption
   
   One challenge is that it would be hard to look at the contents of the metadata table unless we can bring the root table online. At that point, we may be too far into the upgrade process to tell users to go back to 2.1 and compact the relevant tablets using RFiles.
   
   We could also add an "upgrade-readiness tool" to 2.1.x to allow users to check if they are able to upgrade to 3.x.
   
   We could also allow the upgrade to complete, but provide a converter tool to convert a map file to an RFile and perform the metadata update. That might still allow us to remove a lot of the Map file support code.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [accumulo] EdColeman commented on issue #3359: Drop Mapfile support

["EdColeman (via GitHub)" <gi...@apache.org>]

EdColeman commented on issue #3359:
URL: https://github.com/apache/accumulo/issues/3359#issuecomment-1527580974

   If I understand things correctly, It may be possible to read the root tablet outside of the manager and with only ZooKeeper running - the root tablet data is stored as a json blog under /accumulo/x/root_tablet and it is trivial to read / decode that and look for the file names / extensions.  That should be able to provide a pre-start check that at least the root and metadata tables are rfiles and could be brought online.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [accumulo] ctubbsii closed issue #3359: Drop Mapfile support

["ctubbsii (via GitHub)" <gi...@apache.org>]

ctubbsii closed issue #3359: Drop Mapfile support
URL: https://github.com/apache/accumulo/issues/3359


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [accumulo] ctubbsii commented on issue #3359: Drop Mapfile support

["ctubbsii (via GitHub)" <gi...@apache.org>]

ctubbsii commented on issue #3359:
URL: https://github.com/apache/accumulo/issues/3359#issuecomment-1535500498

   MyMapFile (a customized version of Hadoop's MapFile was dropped in https://issues.apache.org/jira/browse/ACCUMULO-288) for 1.5.0. Ever since, the MapFile reading support has been broken because MapFileIterator never set topKey or topValue to anything other than null, so reading from that iterator for an existing MapFile would have made every file look like an empty one.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org