Re: [I] PinotFSSegmentUploader uploads an unbounded number of files when AWS SDK has issues [pinot]

["Jackie-Jiang (via GitHub)" <gi...@apache.org>]

Jackie-Jiang commented on issue #12264:
URL: https://github.com/apache/pinot/issues/12264#issuecomment-1934820014

   I believe the intention of uploading a temp file first, then moving it to the final location is to avoid race condition or failed file upload. Thinking through the problem, I guess we should be able to directly upload the segment to final location. We will need some extra logic to handle the previous failed upload. @mcvsubbu @sajjad-moradi Do you have concerns on this?
   
   Currently the temp files will be cleaned either when the segment commits, or through an async task. See #10815 for more details. One caveat is that if the segment is not committed, the temp files won't be cleaned up.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org