[GitHub] [accumulo] keith-turner commented on pull request #3350: modifies bulk import to use conditional mutations

["keith-turner (via GitHub)" <gi...@apache.org>]

keith-turner commented on PR #3350:
URL: https://github.com/apache/accumulo/pull/3350#issuecomment-1530052719

   > I'm curious if you need a refresh column in the metadata. You could send the RPC to the hosting tablet server as a FaTE transaction and expect a response. If the call times out or fails, then you could retry as part of the FaTE framework.
   
   I think that could be done and it may be more efficient for the normal case when everything is stable and working well.  The refresh column handles the following cases that would not handled by an RPC alone.
   
    * The refresh column is set with the location of the tserver at the time the files were added.  So it can detect if the tablet has moved since it added the files and if it did move then it knows that no refresh is needed.
    * In the case where the manager dies, the refresh column keeps track of what work is left across manager restarts.
    * The transaction id in the refresh column allows the tablet server to know if it has already recently loaded something.  For example if 3 fate transactions all want to refresh a tablet at around the same time the tablet can do one actual refresh and satisfy all request.  Without the refresh column the tablet will always have reload its metadata when requested even if its not needed (because a recent reload it did satisfies the request).
    * The refresh column allows FATE to process the refresh operations in the isReady call using async RPC which means a fate thread is not tied up.  Without the refresh column a FATE thread would have to be tied up until all tablets are refreshed because the tracking of what is refreshed is moved from metadata table to memory.  Also the Fate operation would have to use synchronous RPCs vs async.  Also the fate thread has to keep everything in memory.
   
   Using only an RPC, I think we could do the following.  
   
    * Scan the metadata table for tablets with load markers and build a map of `Map<TServerLoc, List<KeyExtent>>`.  The map may contain some tservers that do not actually need to refresh because they were not the location when the files were set, but that does not hurt anythnig.
    * Continue sending synchronous RPCs unil a positive response has been received for everything in the map. 
   
   Barring all the edge cases, this may be faster. The fate thread would be tied up for however long this takes. If all the fate threads are tied up, more can be added as long as memory supports it.  There are other fate operations that hold a bunch of stuff in memory and/or tie up a fate thread for an indefinite time period while waiting for a condition to become true.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org