[GitHub] [pinot] moszutij commented on issue #8116: A segment file is not removed (moved to Deleted_Segments folder) from an S3 backed segment store when a segment is deleted

[GitBox <gi...@apache.org>]

moszutij commented on issue #8116:
URL: https://github.com/apache/pinot/issues/8116#issuecomment-1029378519


   I think the root cause might be the differences between the implementation of LocalPinotFS.exists(...). and S3PinotFS.exists(...).
   
   LocalPinotFS.exists() for the following URI: file://var/pinot/controller/data/actions/ will return true and then attempt to move the file(s) to the Deleted_Segments folder whereas 
   S3PinotFS.exists() for the following URI: s3://<bucket>/pinot/controller/data/actions/ will return false and then log a warning that it is unable to find the segment file. 
   
   I know that the concept of folders does not really exist in object storage and files in a bucket contain slashes to emulate folders in a file system. That could be the reason but I didn't dig any further.
   
   My use case was to implement a segment retention policy where S3 is used as the backed segment store e.g. delete segments that are older than 1 year which wasn't working for me and was finally narrowed down to this issue.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org