[GitHub] keith-turner commented on issue #449: Seeing more data loss when running continuous ingest with agitation.

[GitBox <gi...@apache.org>]

keith-turner commented on issue #449: Seeing more data loss when running continuous ingest with agitation.
URL: https://github.com/apache/accumulo/issues/449#issuecomment-385106223
 
 
   I determined the cause of this bug.  It happens when a tablet is unloaded from a tserver and then later comes back. When it comes back, if data is written and no minor compaction occurs before the tserver dies then that data may not be seen.  Below is a detailed example, the seq# below are WAL seq#.
   
    * TabletA is loaded on tserver5
    * Define tablet is written to WAL1 with seq#1
    * TabletA is loaded on tserver5 for a bit and many compactions occur
    * Data is written to WAL1 for tabletA with seq# 50
    * TabletA is compacted
      * compaction start is written to WAL with seq# 51
      * compaction finish is writtent to WAL with seq# 52
    * TabletA is unloaded from tserver5
    * tserver5 starts using WAL2, however WAL1 is still referenced by other tablets
    * TabletA is loaded from tserver5
    * Define tablet is written to WAL2 with seq#1
    * Data is written to WAL2 for TabletA with seq# 1 (the seq# is lower because it resets when tablet is reassigned)
    * tserver5 dies
   
   During recovery of TabletA WAL1 and WAL2 are examined to find the last compaction to finish and its seq#.  The last compaction to finish has a seq# of 52, therefore we look for data >= 52.  However the data in WAL2 has a lower seq# and is therefore ignored.
   
   Before 1.8.0 WAL1 would never have been included in the recovery and this problem would not have occurred.  However the changes made in 1.8.0 (moved from tracking WAL refs per tablet to per tablet server) bring WAL1 into the recovery process.  

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services