You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Michael McCandless (JIRA)" <ji...@apache.org> on 2009/12/06 11:01:18 UTC
[jira] Created: (LUCENE-2120) Possible file handle leak in near
real-time reader
Possible file handle leak in near real-time reader
--------------------------------------------------
Key: LUCENE-2120
URL: https://issues.apache.org/jira/browse/LUCENE-2120
Project: Lucene - Java
Issue Type: Bug
Components: Index
Affects Versions: 3.1
Reporter: Michael McCandless
Assignee: Michael McCandless
Fix For: 3.1
Spinoff of LUCENE-1526: Jake/John hit file descriptor exhaustion when testing NRT.
I've tried to repro this, stress testing NRT, saturating reopens, indexing, searching, but haven't found any issue.
Let's try to get to the bottom of it, here...
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2120) Possible file handle leak in near
real-time reader
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12786593#action_12786593 ]
Michael McCandless commented on LUCENE-2120:
--------------------------------------------
Zoie code that's invoking NRT is here: http://code.google.com/p/zoie/source/browse/branches/BR_DELETE_OPT/java/proj/zoie/impl/indexing/luceneNRT/ThrottledLuceneNRTDataConsumer.java
> Possible file handle leak in near real-time reader
> --------------------------------------------------
>
> Key: LUCENE-2120
> URL: https://issues.apache.org/jira/browse/LUCENE-2120
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Affects Versions: 3.1
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Fix For: 3.1
>
>
> Spinoff of LUCENE-1526: Jake/John hit file descriptor exhaustion when testing NRT.
> I've tried to repro this, stress testing NRT, saturating reopens, indexing, searching, but haven't found any issue.
> Let's try to get to the bottom of it, here...
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2120) Possible file handle leak in near
real-time reader
Posted by "John Wang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12794998#action_12794998 ]
John Wang commented on LUCENE-2120:
-----------------------------------
Hi Michael:
You are abs. right! By adding an integer increment in the loop made a difference. From my laptop, I see the break even point at 1.5%.
For what we are using it for in Zoie, the trade-off is worth-while. Because we rely on Lucene to do the delete check, and use this iteration for skipping over transient deletes, e.g. the ones that have not been made to the index. Normally they are << corpus size, e.g. 100 - 1k out of 5M docs. The memory cost for this is also very small in comparison due to the sparsity of del docset.
-John
> Possible file handle leak in near real-time reader
> --------------------------------------------------
>
> Key: LUCENE-2120
> URL: https://issues.apache.org/jira/browse/LUCENE-2120
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Affects Versions: 3.1
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Fix For: 3.1
>
>
> Spinoff of LUCENE-1526: Jake/John hit file descriptor exhaustion when testing NRT.
> I've tried to repro this, stress testing NRT, saturating reopens, indexing, searching, but haven't found any issue.
> Let's try to get to the bottom of it, here...
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2120) Possible file handle leak in near
real-time reader
Posted by "John Wang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12793803#action_12793803 ]
John Wang commented on LUCENE-2120:
-----------------------------------
Yes we have done perf tests.
We see no indexing throughput improvement, query throughput improved by 40%.
> Possible file handle leak in near real-time reader
> --------------------------------------------------
>
> Key: LUCENE-2120
> URL: https://issues.apache.org/jira/browse/LUCENE-2120
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Affects Versions: 3.1
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Fix For: 3.1
>
>
> Spinoff of LUCENE-1526: Jake/John hit file descriptor exhaustion when testing NRT.
> I've tried to repro this, stress testing NRT, saturating reopens, indexing, searching, but haven't found any issue.
> Let's try to get to the bottom of it, here...
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2120) Possible file handle leak in near
real-time reader
Posted by "John Wang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12794618#action_12794618 ]
John Wang commented on LUCENE-2120:
-----------------------------------
I realized in my ArrayDocIdSet.skip, i am doing binary search always between index 0 and array.length-1, after optimizing this, the percentage is at 4% instead of 2%.
One important distinction is the memory consumption, the skip algorithm is much compact when the deleted set is sparse compare to the corpus.
> Possible file handle leak in near real-time reader
> --------------------------------------------------
>
> Key: LUCENE-2120
> URL: https://issues.apache.org/jira/browse/LUCENE-2120
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Affects Versions: 3.1
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Fix For: 3.1
>
>
> Spinoff of LUCENE-1526: Jake/John hit file descriptor exhaustion when testing NRT.
> I've tried to repro this, stress testing NRT, saturating reopens, indexing, searching, but haven't found any issue.
> Let's try to get to the bottom of it, here...
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2120) Possible file handle leak in near
real-time reader
Posted by "John Wang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12790985#action_12790985 ]
John Wang commented on LUCENE-2120:
-----------------------------------
bq. is this what the private static int MAX_READER_GENERATION = 3
No, I misunderstood you. This is just a number I think it is safe to keep for make sure you don't close reader while it is being searched.
bq. you resolve the deleted docs in the BG
That is not really true. It is stored in a DocIdSet, and it does the skipping in a special TermDocs for avoid deletes.
bq. It's sort of a "warm in the background" tradeoff, ie, give me my reader very quickly, even if the first searches against it must run a bit slower since they double check deletions, until the warming is done.... vs Lucene which forcefully "warms" (making reopen time longer) before returning the reader to you.
I am not sure I understand what you mean by "double check deletions". Warming is done in the background, in the foreground you do search along with ram indexes that hold transient indexing updates. So one guarantee is that the any time search is happening the disk index is warm, so you don't have to pay the cost related to warming.
BTW, i have merged BR_DELETE_OPT down to trunk and did a release. Feel free to take from trunk.
> Possible file handle leak in near real-time reader
> --------------------------------------------------
>
> Key: LUCENE-2120
> URL: https://issues.apache.org/jira/browse/LUCENE-2120
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Affects Versions: 3.1
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Fix For: 3.1
>
>
> Spinoff of LUCENE-1526: Jake/John hit file descriptor exhaustion when testing NRT.
> I've tried to repro this, stress testing NRT, saturating reopens, indexing, searching, but haven't found any issue.
> Let's try to get to the bottom of it, here...
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2120) Possible file handle leak in near
real-time reader
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12793775#action_12793775 ]
Michael McCandless commented on LUCENE-2120:
--------------------------------------------
I briefly diff'd the BR_DELETE_OPT branch in Zoie -- it looks like you switched (via custom TermDocs/Positions) to enforcing deleted docs using an iterator instead of double-check random-access, right? I had tried the same thing with Lucene (a while back now), under LUCENE-1476/LUCENE-1536, and found that the simple BitVector gave much better performance than an iterator (which then led to applying filters random-access as well). Have you tested performance of this switch? Maybe it works out to be faster, net/net, than doing the double-deletions check?
> Possible file handle leak in near real-time reader
> --------------------------------------------------
>
> Key: LUCENE-2120
> URL: https://issues.apache.org/jira/browse/LUCENE-2120
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Affects Versions: 3.1
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Fix For: 3.1
>
>
> Spinoff of LUCENE-1526: Jake/John hit file descriptor exhaustion when testing NRT.
> I've tried to repro this, stress testing NRT, saturating reopens, indexing, searching, but haven't found any issue.
> Let's try to get to the bottom of it, here...
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2120) Possible file handle leak in near
real-time reader
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12786600#action_12786600 ]
Michael McCandless commented on LUCENE-2120:
--------------------------------------------
Why does Zoie even retain 3 readers? Why not keep only the current one?
> Possible file handle leak in near real-time reader
> --------------------------------------------------
>
> Key: LUCENE-2120
> URL: https://issues.apache.org/jira/browse/LUCENE-2120
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Affects Versions: 3.1
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Fix For: 3.1
>
>
> Spinoff of LUCENE-1526: Jake/John hit file descriptor exhaustion when testing NRT.
> I've tried to repro this, stress testing NRT, saturating reopens, indexing, searching, but haven't found any issue.
> Let's try to get to the bottom of it, here...
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2120) Possible file handle leak in near
real-time reader
Posted by "John Wang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12790506#action_12790506 ]
John Wang commented on LUCENE-2120:
-----------------------------------
Hi Michael:
bq: Why does Zoie even retain 3 readers? Why not keep only the current one?
1 mem reader for when the disk batch, 1 mem reader for the time disk reader indexes, 1 disk reader
bq: It looks like the test uses both Wikipedia & Medline for document sources? Do I really need both?
By default, it only runs with Medline data. You don't need both. perf/settings/index.properties->data.type dictates which to use, file->medline, wiki->wikipedia
Also,
You should use the branch: BR_DELETE_OPT
It has the optimization you suggested on handling deleted docs, e.g. should not check for each hit candidate with IntSetAccelerator.
Also, I have added a DataConsumer to handle delayed reopen for NRT case. You see the file handle leakage quickly with it: see perf/conf/zoie.properties to turn on ThrottledLuceneNRTDataConsumer.
On my mac, I use lsof to see the file handle count.
-John
> Possible file handle leak in near real-time reader
> --------------------------------------------------
>
> Key: LUCENE-2120
> URL: https://issues.apache.org/jira/browse/LUCENE-2120
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Affects Versions: 3.1
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Fix For: 3.1
>
>
> Spinoff of LUCENE-1526: Jake/John hit file descriptor exhaustion when testing NRT.
> I've tried to repro this, stress testing NRT, saturating reopens, indexing, searching, but haven't found any issue.
> Let's try to get to the bottom of it, here...
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2120) Possible file handle leak in near
real-time reader
Posted by "John Wang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12794617#action_12794617 ]
John Wang commented on LUCENE-2120:
-----------------------------------
Michael:
I wrote a little test to measure and to understand the perf:
I compare the two methods:
static long testBits(BitVector bv,int numhits) throws Exception{
long start = System.nanoTime();
for (int i=0;i<numhits;++i){
if (bv.get(i)){
}
}
long end = System.nanoTime();
return (end-start);
}
static long testSkipIter(ArrayDocIdSet delSet,int numhits) throws Exception{
DocIdSetIterator delIter = delSet.iterator();
int nextDelDoc = delIter.nextDoc();
long start = System.nanoTime();
for (int i=0;i<numhits;++i){
if (i>=nextDelDoc){
if (i==nextDelDoc){
}
nextDelDoc = delIter.advance(i);
}
}
long end = System.nanoTime();
return (end-start);
}
I removed everything to the barebones to understand the perf implications.
Here are the results on my macbook pro, with numHits and del count:
5M 500:
bits: 42417850
skip: 15234650
5M 100:
bits: 43053650
skip: 15268850
5M 10k:
bits: 41694350
skip: 17504900
5M 100k:
bits: 41737350
skip: 42966000
1M 1000:
bits: 8722700
skip: 3249100
1M 10k:
bits: 8210650
skip: 6119700
1M 25k:
bits: 8558150
skip: 9477850
You see BitVector starts to win with numDeletes's density at about 2%, and it is pretty consistent between diff numHits parameter.
So in real life scenario, we see that the numDeletes to be very small. However, it would be a great improvement if we can case it out depending on result set.
> Possible file handle leak in near real-time reader
> --------------------------------------------------
>
> Key: LUCENE-2120
> URL: https://issues.apache.org/jira/browse/LUCENE-2120
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Affects Versions: 3.1
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Fix For: 3.1
>
>
> Spinoff of LUCENE-1526: Jake/John hit file descriptor exhaustion when testing NRT.
> I've tried to repro this, stress testing NRT, saturating reopens, indexing, searching, but haven't found any issue.
> Let's try to get to the bottom of it, here...
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2120) Possible file handle leak in near
real-time reader
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12791065#action_12791065 ]
Michael McCandless commented on LUCENE-2120:
--------------------------------------------
{quote}
bq. is this what the private static int MAX_READER_GENERATION = 3
No, I misunderstood you. This is just a number I think it is safe to keep for make sure you don't close reader while it is being searched.
{quote}
Ahh I see.... you can incRef/decRef the reader to handle that.
{quote}
bq. you resolve the deleted docs in the BG
That is not really true. It is stored in a DocIdSet, and it does the skipping in a special TermDocs for avoid deletes.
I am not sure I understand what you mean by "double check deletions". Warming is done in the background, in the foreground you do search along with ram indexes that hold transient indexing updates. So one guarantee is that the any time search is happening the disk index is warm, so you don't have to pay the cost related to warming.
{quote}
Sorry, maybe I'm misunderstanding what was done on the BR_DELETE_OPT branch -- I thought the idea was to avoid having to always double-check deletes by resolving them to docIDs in the BG (described at https://issues.apache.org/jira/browse/LUCENE-1526?focusedCommentId=12776994&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12776994). I'll try to diff the branch...
bq. BTW, i have merged BR_DELETE_OPT down to trunk and did a release. Feel free to take from trunk.
OK thanks.
> Possible file handle leak in near real-time reader
> --------------------------------------------------
>
> Key: LUCENE-2120
> URL: https://issues.apache.org/jira/browse/LUCENE-2120
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Affects Versions: 3.1
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Fix For: 3.1
>
>
> Spinoff of LUCENE-1526: Jake/John hit file descriptor exhaustion when testing NRT.
> I've tried to repro this, stress testing NRT, saturating reopens, indexing, searching, but haven't found any issue.
> Let's try to get to the bottom of it, here...
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2120) Possible file handle leak in near
real-time reader
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12790260#action_12790260 ]
Michael McCandless commented on LUCENE-2120:
--------------------------------------------
Jake/John, how can I run the Zoie test that hits the file descriptor / memory / incorrect deletions with Lucene's NRT? Is it some variant of this?
http://code.google.com/p/zoie/wiki/Running_Performance_Test
It looks like the test uses both Wikipedia & Medline for document sources? Do I really need both?
> Possible file handle leak in near real-time reader
> --------------------------------------------------
>
> Key: LUCENE-2120
> URL: https://issues.apache.org/jira/browse/LUCENE-2120
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Affects Versions: 3.1
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Fix For: 3.1
>
>
> Spinoff of LUCENE-1526: Jake/John hit file descriptor exhaustion when testing NRT.
> I've tried to repro this, stress testing NRT, saturating reopens, indexing, searching, but haven't found any issue.
> Let's try to get to the bottom of it, here...
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2120) Possible file handle leak in near
real-time reader
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12790673#action_12790673 ]
Michael McCandless commented on LUCENE-2120:
--------------------------------------------
Thanks for the details John!
{quote}
bq: Why does Zoie even retain 3 readers? Why not keep only the current one?
1 mem reader for when the disk batch, 1 mem reader for the time disk reader indexes, 1 disk reader
{quote}
Hmm -- is this what the {{private static int MAX_READER_GENERATION = 3;}} in ThrottledLuceneNRTDataConsumer (link above) is doing? From that code, it looks like it just always retains the last 3 reopened readers... it sounds like the logic to keep the 2 mem readers & 1 disk reader is elsewhere (in addition) in Zoie? Or maybe this logic is what's doing that? (Not yet familiar enough w/ Zoie...).
bq. By default, it only runs with Medline data. You don't need both. perf/settings/index.properties->data.type dictates which to use, file->medline, wiki->wikipedia
OK I'll try to use only Wikipedia -- I've already slurped that down. It'd be great to get a ContentSource in contrib/benchmark that can produce docs from medline...
{quote}
You should use the branch: BR_DELETE_OPT
It has the optimization you suggested on handling deleted docs, e.g. should not check for each hit candidate with IntSetAccelerator.
Also, I have added a DataConsumer to handle delayed reopen for NRT case. You see the file handle leakage quickly with it: see perf/conf/zoie.properties to turn on ThrottledLuceneNRTDataConsumer.
On my mac, I use lsof to see the file handle count.
{quote}
OK, will do. So (on this branch) you resolve the deleted docs in the BG, so that, once finished, the reader no longer double-checks hits for deletions? That sounds like a good improvement...
It's sort of a "warm in the background" tradeoff, ie, give me my reader very quickly, even if the first searches against it must run a bit slower since they double check deletions, until the warming is done.... vs Lucene which forcefully "warms" (making reopen time longer) before returning the reader to you.
> Possible file handle leak in near real-time reader
> --------------------------------------------------
>
> Key: LUCENE-2120
> URL: https://issues.apache.org/jira/browse/LUCENE-2120
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Affects Versions: 3.1
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Fix For: 3.1
>
>
> Spinoff of LUCENE-1526: Jake/John hit file descriptor exhaustion when testing NRT.
> I've tried to repro this, stress testing NRT, saturating reopens, indexing, searching, but haven't found any issue.
> Let's try to get to the bottom of it, here...
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2120) Possible file handle leak in near
real-time reader
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12794819#action_12794819 ]
Michael McCandless commented on LUCENE-2120:
--------------------------------------------
Thanks for running that test John. You should probably add something
inside the if bodies, eg increment a count, to make sure the compiler
doesn't optimize it away. (And print the count in the end to make
sure it's identical for the two methods).
I had done a similar test in LUCENE-1476, where I hacked
SegmentTermDocs to represent deleted docs as list-of-int-docIDs, and
then "iterate" through them, but found only a tiny perf. gain at <= 1%
deleted docs. Not sure why I didn't see a bigger gain... I would
expect at low rates to see gains.
And the perf. loss at even moderate deletion rates is what inspired
opening LUCENE-1536, to explore applying filters the same way we apply
deleted docs (ie, switch from iterator to random access).
> Possible file handle leak in near real-time reader
> --------------------------------------------------
>
> Key: LUCENE-2120
> URL: https://issues.apache.org/jira/browse/LUCENE-2120
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Affects Versions: 3.1
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Fix For: 3.1
>
>
> Spinoff of LUCENE-1526: Jake/John hit file descriptor exhaustion when testing NRT.
> I've tried to repro this, stress testing NRT, saturating reopens, indexing, searching, but haven't found any issue.
> Let's try to get to the bottom of it, here...
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org