You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2022/09/16 18:09:41 UTC
[GitHub] [pinot] TT1103 opened a new issue, #9415: Minion Keeping segments on disk after RealtimeToOfflineSegmentsTask
TT1103 opened a new issue, #9415:
URL: https://github.com/apache/pinot/issues/9415
We noticed high disk usage on the minion servers and it appears that the RealtimeToOfflineSegmentsTask sometimes leaves the segments on disk.
We also noticed that Pinot is keeping a large amount deleted files open, which might be a memory management bug.
Example `lsof -n | grep deleted` output:
```
java 106586 8183 TaskState username 634w REG 252,0 7401250816 5887067 /tmp/PinotMinion/data/RealtimeToOfflineSegmentsTask/tmp-6fe9e9c9-238c-4b99-8018-094f32bbad38/workingDir/mapper_output/0/record.offset (deleted)
java 106586 8183 TaskState username 635w REG 252,0 128791597056 5887068 /tmp/PinotMinion/data/RealtimeToOfflineSegmentsTask/tmp-6fe9e9c9-238c-4b99-8018-094f32bbad38/workingDir/mapper_output/0/record.data (deleted)
java 106586 8184 HelixTask username 634w REG 252,0 7401250816 5887067 /tmp/PinotMinion/data/RealtimeToOfflineSegmentsTask/tmp-6fe9e9c9-238c-4b99-8018-094f32bbad38/workingDir/mapper_output/0/record.offset (deleted)
```
see slack thread: https://apache-pinot.slack.com/archives/C011C9JHN7R/p1663342366477189
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] walterddr closed issue #9415: Minion Keeping segments on disk after RealtimeToOfflineSegmentsTask
Posted by GitBox <gi...@apache.org>.
walterddr closed issue #9415: Minion Keeping segments on disk after RealtimeToOfflineSegmentsTask
URL: https://github.com/apache/pinot/issues/9415
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #9415: Minion Keeping segments on disk after RealtimeToOfflineSegmentsTask
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on issue #9415:
URL: https://github.com/apache/pinot/issues/9415#issuecomment-1251624307
@klsince Can you help take a look at this issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] snleee commented on issue #9415: Minion Keeping segments on disk after RealtimeToOfflineSegmentsTask
Posted by GitBox <gi...@apache.org>.
snleee commented on issue #9415:
URL: https://github.com/apache/pinot/issues/9415#issuecomment-1317428601
Good catch 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] klsince commented on issue #9415: Minion Keeping segments on disk after RealtimeToOfflineSegmentsTask
Posted by GitBox <gi...@apache.org>.
klsince commented on issue #9415:
URL: https://github.com/apache/pinot/issues/9415#issuecomment-1262928527
some quick thoughts but need verify: although we do delete the files in a finally{} upon task completion, but we may not release all the mmap over those files, which is done by
```
// in SegmentProcessorFramework.process(): L145, under the hood, the record.offset/data files are mmap'ed
GenericRowFileReader fileReader = fileManager.getFileReader();
...
// in SegmentProcessorFramework.process(): L166, but this is not called in finally block
fileManager.cleanUp();
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org