Re: [PR] Hadoop segment index file rename (druid)

["gianm (via GitHub)" <gi...@apache.org>]

gianm commented on PR #11194:
URL: https://github.com/apache/druid/pull/11194#issuecomment-1845655509

   Some additional color- this was done to fix a race condition that was suspected to be something with `s3` deep storage (using Hadoop's `s3n` adapter) that goes like:
   
   1. Reducer A launches
   2. Reducer A successfully uploads a segment to temp location
   3. Reducer A starts renaming the temp segment to the final location (this is really a delete of the final location if it exists, then a copy of the temp to the final, and finally deletion of the temp)
   4. Reducer A takes too long, triggering a replica Reducer B to be launched speculatively
   5. While Reducer A is still copying the temp to the final, Reducer B finishes ingesting and uploads its temp file.
   6. While Reducer A is still copying the temp to the final, Reducer B issues a delete for the final file being written by A, because  is "renaming" its temp file
   7. Reducer A finishes its rename
   8. Deletion issued by Reducer B in step 6 takes effect and deletes the final segment
   9. Reducer A completes, because it finished the rename
   10. Reducer A finishes, so it's the "winning" speculative execution replica, and Reducer B gets terminated before it can finish moving its temp file to the final location, leaving just its temp file (is this how Hadoop speculative execution works?)
   
   The intent was to fix a problem where sometimes, instead of `index.zip` being present in the final location, it's `index.zip.1` or something like that. We do believe this fixed the problem, although the cost is that all the renames got moved to the launcher `index_hadoop` job rather than being spread throughout the various reducers. It would potentially slow down jobs that create large numbers of segments using large numbers of reducers.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org