You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2021/02/09 17:19:36 UTC
[GitHub] [incubator-pinot] sajjad-moradi opened a new issue #6562: Real time tables with text index enabled lead to file leaks
sajjad-moradi opened a new issue #6562:
URL: https://github.com/apache/incubator-pinot/issues/6562
When a consuming segment completes, all the files in consuming directory are supposed to be deleted. However, text index for real time tables does not clean up all the files. For each segment, one file will remain undeleted.
I dug a bit into the code and I believe found the root cause.
In “consuming” -> “online” transition on server, first we delete the consuming segment directory and then release the segment:
Text index uses a file “write.lock” as the lock. The lock is released whenever the text index gets released.
So when we try to delete the consuming segment directory in CONSUMING to ONLINE transition, since the lock file is being used and not yet released, all files in segment directory get deleted except the lock file.
That’s why you see the following on a server having a realtime table with text index enabled:
The size of those lock files are all zero:
```bash
[smoradi@lva1-app0232 ~]$ ll /export/content/data/pinot-server/i001/pinot/consumerDir/RealtimeFeatureTest1_REALTIME/RealtimeFeatureTest1__0__101__20201229T1103Z/textDim1.lucene.index/
total 0
-rw-rw-r-- 1 app app 0 Dec 29 11:03 write.lock
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] pabrahamusa edited a comment on issue #6562: Real time tables with text index enabled lead to file leaks
Posted by GitBox <gi...@apache.org>.
pabrahamusa edited a comment on issue #6562:
URL: https://github.com/apache/incubator-pinot/issues/6562#issuecomment-778567830
I can also see lots of these lock file not getting deleted. @sajjad-moradi , @mcvsubbu any fix you guys planning to add?
```
rw-r--r-- 1 root root 0 Feb 13 00:01 ./consumers/sblog__1__753__20210213T0000Z/log.lucene.index/write.lock
-rw-r--r-- 1 root root 0 Feb 13 00:02 ./consumers/sblog__1__754__20210213T0002Z/log.lucene.index/write.lock
-rw-r--r-- 1 root root 0 Feb 13 00:04 ./consumers/sblog__1__755__20210213T0004Z/log.lucene.index/write.lock
-rw-r--r-- 1 root root 0 Feb 13 00:06 ./consumers/sblog__1__756__20210213T0006Z/log.lucene.index/write.lock
-rw-r--r-- 1 root root 0 Feb 13 00:08 ./consumers/sblog__1__757__20210213T0008Z/log.lucene.index/write.lock
-rw-r--r-- 1 root root 0 Feb 13 00:12 ./consumers/sblog__1__758__20210213T0012Z/log.lucene.index/write.lock
-rw-r--r-- 1 root root 0 Feb 13 00:13 ./consumers/sblog__1__759__20210213T0013Z/log.lucene.index/write.lock
-rw-r--r-- 1 root root 0 Feb 13 00:16 ./consumers/sblog__1__760__20210213T0016Z/log.lucene.index/write.lock
-rw-r--r-- 1 root root 0 Feb 13 00:17 ./consumers/sblog__1__761__20210213T0017Z/log.lucene.index/write.lock
-rw-r--r-- 1 root root 0 Feb 13 00:19 ./consumers/sblog__1__762__20210213T0019Z/log.lucene.index/write.lock
-rw-r--r-- 1 root root 0 Feb 13 00:21 ./consumers/sblog__1__763__20210213T0021Z/log.lucene.index/write.lock
-rw-r--r-- 1 root root 0 Feb 13 00:22 ./consumers/sblog__1__764__20210213T0022Z/log.lucene.index/write.lock
-rw-r--r-- 1 root root 0 Feb 13 00:26 ./consumers/sblog__1__765__20210213T0026Z/log.lucene.index/write.lock
-rw-r--r-- 1 root root 0 Feb 13 00:28 ./consumers/sblog__1__766__20210213T0028Z/log.lucene.index/write.lock
-rw-r--r-- 1 root root 0 Feb 13 00:31 ./consumers/sblog__1__767__20210213T0031Z/log.lucene.index/write.lock
-rw-r--r-- 1 root root 0 Feb 13 00:33 ./consumers/sblog__1__768__20210213T0033Z/log.lucene.index/write.lock
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] kishoreg commented on issue #6562: Real time tables with text index enabled lead to file leaks
Posted by GitBox <gi...@apache.org>.
kishoreg commented on issue #6562:
URL: https://github.com/apache/incubator-pinot/issues/6562#issuecomment-776101408
@sajjad-moradi awesome find
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] pabrahamusa commented on issue #6562: Real time tables with text index enabled lead to file leaks
Posted by GitBox <gi...@apache.org>.
pabrahamusa commented on issue #6562:
URL: https://github.com/apache/incubator-pinot/issues/6562#issuecomment-778567830
I can also see lots of these lock file not getting deleted. @sajjad-moradi , @mcvsubbu and fix you guys planning to add?
```
rw-r--r-- 1 root root 0 Feb 13 00:01 ./consumers/sblog__1__753__20210213T0000Z/log.lucene.index/write.lock
-rw-r--r-- 1 root root 0 Feb 13 00:02 ./consumers/sblog__1__754__20210213T0002Z/log.lucene.index/write.lock
-rw-r--r-- 1 root root 0 Feb 13 00:04 ./consumers/sblog__1__755__20210213T0004Z/log.lucene.index/write.lock
-rw-r--r-- 1 root root 0 Feb 13 00:06 ./consumers/sblog__1__756__20210213T0006Z/log.lucene.index/write.lock
-rw-r--r-- 1 root root 0 Feb 13 00:08 ./consumers/sblog__1__757__20210213T0008Z/log.lucene.index/write.lock
-rw-r--r-- 1 root root 0 Feb 13 00:12 ./consumers/sblog__1__758__20210213T0012Z/log.lucene.index/write.lock
-rw-r--r-- 1 root root 0 Feb 13 00:13 ./consumers/sblog__1__759__20210213T0013Z/log.lucene.index/write.lock
-rw-r--r-- 1 root root 0 Feb 13 00:16 ./consumers/sblog__1__760__20210213T0016Z/log.lucene.index/write.lock
-rw-r--r-- 1 root root 0 Feb 13 00:17 ./consumers/sblog__1__761__20210213T0017Z/log.lucene.index/write.lock
-rw-r--r-- 1 root root 0 Feb 13 00:19 ./consumers/sblog__1__762__20210213T0019Z/log.lucene.index/write.lock
-rw-r--r-- 1 root root 0 Feb 13 00:21 ./consumers/sblog__1__763__20210213T0021Z/log.lucene.index/write.lock
-rw-r--r-- 1 root root 0 Feb 13 00:22 ./consumers/sblog__1__764__20210213T0022Z/log.lucene.index/write.lock
-rw-r--r-- 1 root root 0 Feb 13 00:26 ./consumers/sblog__1__765__20210213T0026Z/log.lucene.index/write.lock
-rw-r--r-- 1 root root 0 Feb 13 00:28 ./consumers/sblog__1__766__20210213T0028Z/log.lucene.index/write.lock
-rw-r--r-- 1 root root 0 Feb 13 00:31 ./consumers/sblog__1__767__20210213T0031Z/log.lucene.index/write.lock
-rw-r--r-- 1 root root 0 Feb 13 00:33 ./consumers/sblog__1__768__20210213T0033Z/log.lucene.index/write.lock
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org