You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "dang-stripe (via GitHub)" <gi...@apache.org> on 2023/06/21 17:22:08 UTC
[GitHub] [pinot] dang-stripe opened a new issue, #10956: Segments in Deleted_Segments do not get retentioned
dang-stripe opened a new issue, #10956:
URL: https://github.com/apache/pinot/issues/10956
We've noticed segments in S3 are not getting deleted from the Deleted_Segments directory. This seems to be because when SegmentDeletionManager calls S3PinotFS.listFiles, it only returns files and not directories and always returns an empty list, causing the `removeAgedDeletedSegments` call to terminate. The Deleted_Segments directory only contains a directory for each table w/ deleted segments.
We found this log line that confirms this which reports 0 files being returned.
```
[2023-06-21 17:02:32.807244] INFO [S3PinotFS] [pool-17-thread-1:169] Listed 0 files from URI: s3://some-bucket/pinot/pinot-cluster-1/prod-1/Deleted_Segments, is recursive: false
```
https://github.com/apache/pinot/blob/master/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/SegmentDeletionManager.java#L290
https://github.com/apache/pinot/blob/master/pinot-plugins/pinot-file-system/pinot-s3/src/main/java/org/apache/pinot/plugin/filesystem/S3PinotFS.java#L486-L499
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #10956: S3 segments in Deleted_Segments do not get retentioned
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #10956:
URL: https://github.com/apache/pinot/issues/10956#issuecomment-1603743787
@snleee Can you help take a look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org