You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "snleee (via GitHub)" <gi...@apache.org> on 2023/09/20 07:49:23 UTC
[GitHub] [pinot] snleee opened a new issue, #11633: Remove the `pinotFS.listFiles()` from the core commit protocol path
snleee opened a new issue, #11633:
URL: https://github.com/apache/pinot/issues/11633
`pinotFS.listFiles()` can be expensive when we use remote file system (e.g. AWS s3) and the number of segments are very high (e.g. 100k). If the number of partition is high, it makes the problem even worse. We have seen the case where `pinotFS.listFiles()` are mostly consuming the threads available for API calls.
Instead of fetching the full list and filter on `<segmentName>.tmp.` to detect the file to delete, we should keep the temp name and directly try to delete the file based on that.
```
try {
for (String uri : pinotFS.listFiles(tableDirURI, false)) {
if (uri.contains(SegmentCompletionUtils.getSegmentNamePrefix(segmentName))) {
LOGGER.warn("Deleting temporary segment file: {}", uri);
Preconditions.checkState(pinotFS.delete(new URI(uri), true), "Failed to delete file: %s", uri);
}
}
} catch (Exception e) {
LOGGER.warn("Caught exception while deleting temporary segment files for segment: {}", segmentName, e);}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] snleee closed issue #11633: Remove the `pinotFS.listFiles()` from the core commit protocol path
Posted by "snleee (via GitHub)" <gi...@apache.org>.
snleee closed issue #11633: Remove the `pinotFS.listFiles()` from the core commit protocol path
URL: https://github.com/apache/pinot/issues/11633
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] ankitsultana commented on issue #11633: Remove the `pinotFS.listFiles()` from the core commit protocol path
Posted by "ankitsultana (via GitHub)" <gi...@apache.org>.
ankitsultana commented on issue #11633:
URL: https://github.com/apache/pinot/issues/11633#issuecomment-1728340732
@snleee : We had hit this exact issue and we just recently did this: https://github.com/apache/pinot/pull/10815
Here's the sort of design doc I had raised a while back https://docs.google.com/document/d/18PqeN7eUP9Yc2MqwihSrGMmadJ3nuDsrcWf6F_ks_hQ/edit#heading=h.ugwy33o0s0l7
Here's the latest code: https://github.com/apache/pinot/blob/9bcbef80ccfb9ae4379f3909c012317ea9cf0921/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java#L469-L481
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] snleee commented on issue #11633: Remove the `pinotFS.listFiles()` from the core commit protocol path
Posted by "snleee (via GitHub)" <gi...@apache.org>.
snleee commented on issue #11633:
URL: https://github.com/apache/pinot/issues/11633#issuecomment-1729967762
marking as closed as this is already addressed as @ankitsultana mentioned. https://github.com/apache/pinot/pull/10815
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org