You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Yonik Seeley (JIRA)" <ji...@apache.org> on 2012/04/21 20:46:34 UTC
[jira] [Created] (SOLR-3392) searcher leak
Yonik Seeley created SOLR-3392:
----------------------------------
Summary: searcher leak
Key: SOLR-3392
URL: https://issues.apache.org/jira/browse/SOLR-3392
Project: Solr
Issue Type: Bug
Reporter: Yonik Seeley
It appears we have a searcher leak where SolrIndexSearcher instances are sometimes not being closed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-3392) searcher leak
Posted by "Dawid Weiss (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13258916#comment-13258916 ]
Dawid Weiss commented on SOLR-3392:
-----------------------------------
Are you talking about the "searchers opened: N, closed: N - Y" errors? This has been relatively frequent but I couldn't reproduce.
> searcher leak
> -------------
>
> Key: SOLR-3392
> URL: https://issues.apache.org/jira/browse/SOLR-3392
> Project: Solr
> Issue Type: Bug
> Reporter: Yonik Seeley
>
> It appears we have a searcher leak where SolrIndexSearcher instances are sometimes not being closed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-3392) searcher leak when
openSearcher=false
Posted by "Dawid Weiss (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13259159#comment-13259159 ]
Dawid Weiss commented on SOLR-3392:
-----------------------------------
https://builds.apache.org/job/Lucene-Solr-tests-only-trunk/13383/consoleText
Just a note -- a recent build with "SolrIndexSearcher opens=74 closes=73" mismatch (suite level).
> searcher leak when openSearcher=false
> -------------------------------------
>
> Key: SOLR-3392
> URL: https://issues.apache.org/jira/browse/SOLR-3392
> Project: Solr
> Issue Type: Bug
> Reporter: Yonik Seeley
> Fix For: 4.0
>
>
> It appears we have a searcher leak where SolrIndexSearcher instances are sometimes not being closed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-3392) searcher leak
Posted by "Dawid Weiss (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13258920#comment-13258920 ]
Dawid Weiss commented on SOLR-3392:
-----------------------------------
bq. and why none of our tests catch it
But they do? There were a number of failures when the number of closes/opens mismatched and failed the build. Maybe I don't understand what you mean.
> searcher leak
> -------------
>
> Key: SOLR-3392
> URL: https://issues.apache.org/jira/browse/SOLR-3392
> Project: Solr
> Issue Type: Bug
> Reporter: Yonik Seeley
>
> It appears we have a searcher leak where SolrIndexSearcher instances are sometimes not being closed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-3392) searcher leak
Posted by "Dawid Weiss (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13258924#comment-13258924 ]
Dawid Weiss commented on SOLR-3392:
-----------------------------------
This one is quite old, I just found it first --
{noformat}
build 14-Apr-2012 06:17:47 [junit] 149810 T1 oas.SolrTestCaseJ4.endTrackingSearchers SEVERE ERROR: SolrIndexSearcher opens=70 closes=69
{noformat}
I always assumed this is something to worry about but never could reproduce it with the same seed.
> searcher leak
> -------------
>
> Key: SOLR-3392
> URL: https://issues.apache.org/jira/browse/SOLR-3392
> Project: Solr
> Issue Type: Bug
> Reporter: Yonik Seeley
>
> It appears we have a searcher leak where SolrIndexSearcher instances are sometimes not being closed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-3392) searcher leak
Posted by "Dawid Weiss (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13258932#comment-13258932 ]
Dawid Weiss commented on SOLR-3392:
-----------------------------------
bq. Those normally don't fail on my box - just our Jenkins box which is really slow +
Ok, thanks for clarification. Those I mentioned happened on our server where I was running Lucene tests for some time too (and that isn't slow). But maybe they were side effects of something else.
> searcher leak
> -------------
>
> Key: SOLR-3392
> URL: https://issues.apache.org/jira/browse/SOLR-3392
> Project: Solr
> Issue Type: Bug
> Reporter: Yonik Seeley
>
> It appears we have a searcher leak where SolrIndexSearcher instances are sometimes not being closed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-3392) searcher leak
Posted by "Yonik Seeley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13258919#comment-13258919 ]
Yonik Seeley commented on SOLR-3392:
------------------------------------
While trying to reproduce the file descriptor leak that Sami and others have reported, I went to
http://localhost:8983/solr/#/singlecore/plugins/core
and noticed that I had 4 open SolrIndexSearcher instances - which means we have a searcher leak somehow.
I'm still trying to track down exactly what's causing it, and why none of our tests catch it (and all of our tests check the number of opens/closes of SolrIndexSearcher instances)
> searcher leak
> -------------
>
> Key: SOLR-3392
> URL: https://issues.apache.org/jira/browse/SOLR-3392
> Project: Solr
> Issue Type: Bug
> Reporter: Yonik Seeley
>
> It appears we have a searcher leak where SolrIndexSearcher instances are sometimes not being closed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-3392) searcher leak when
openSearcher=false
Posted by "Dawid Weiss (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13259174#comment-13259174 ]
Dawid Weiss commented on SOLR-3392:
-----------------------------------
bq. Perhaps there's a way to tell if a test has already failed, then this open/closed searcher test can be skipped?
What Robert pointed to is ok. The "nicest" way to cleanup-on-success only is to write a test rule and cleanup/ make assertions if the sub-statement passes without throwing an exception. @Before*/After* hooks are always executed, regardless of the test output (failure or no failure).
I realize the number of nested stack entries from rules can bother some, but it is really a clean separation of concerns and the ordering is then explicit (both in the code and at runtime) which is nice.
I'm not asking you to fix this -- I will be biting into LuceneTestCase at some point (have some urgent stuff piled up right now) and I will try to clean up @Before/@After* hooks as well so that they're not a single blob of various checks and things.
> searcher leak when openSearcher=false
> -------------------------------------
>
> Key: SOLR-3392
> URL: https://issues.apache.org/jira/browse/SOLR-3392
> Project: Solr
> Issue Type: Bug
> Reporter: Yonik Seeley
> Fix For: 4.0
>
>
> It appears we have a searcher leak where SolrIndexSearcher instances are sometimes not being closed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Resolved] (SOLR-3392) searcher leak
Posted by "Yonik Seeley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yonik Seeley resolved SOLR-3392.
--------------------------------
Resolution: Fixed
Fix Version/s: 4.0
I just committed a fix - it was a really simple silly bug. When openSearcher=false, we requested a new searcher but then never decremented it when we were done. Although the getSearcher() code itself had good test coverage (that's the more complicated code I was worried about), the caller in this specific case (DirectUpdateHandler2) did not.
> searcher leak
> -------------
>
> Key: SOLR-3392
> URL: https://issues.apache.org/jira/browse/SOLR-3392
> Project: Solr
> Issue Type: Bug
> Reporter: Yonik Seeley
> Fix For: 4.0
>
>
> It appears we have a searcher leak where SolrIndexSearcher instances are sometimes not being closed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-3392) searcher leak when
openSearcher=false
Posted by "Yonik Seeley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13259164#comment-13259164 ]
Yonik Seeley commented on SOLR-3392:
------------------------------------
{code}
[junit4] > (@AfterClass output)
[junit4] 2> 35484 T1048 oash.SnapPuller.fetchLatestIndex SEVERE Master at: http://localhost:40685/solr/replication is not available. Index fetch failed. Exception: Connect to localhost:40685 timed out
[junit4] 2> 153931 T846 oas.SolrTestCaseJ4.endTrackingSearchers SEVERE ERROR: SolrIndexSearcher opens=74 closes=73
{code}
It's often the case that an earlier failure can cause tests to abort in such ways that not everything is cleaned up and you'll also see a searcher mismatch error.
Perhaps there's a way to tell if a test has already failed, then this open/closed searcher test can be skipped?
> searcher leak when openSearcher=false
> -------------------------------------
>
> Key: SOLR-3392
> URL: https://issues.apache.org/jira/browse/SOLR-3392
> Project: Solr
> Issue Type: Bug
> Reporter: Yonik Seeley
> Fix For: 4.0
>
>
> It appears we have a searcher leak where SolrIndexSearcher instances are sometimes not being closed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (SOLR-3392) searcher leak when openSearcher=false
Posted by "Yonik Seeley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yonik Seeley updated SOLR-3392:
-------------------------------
Summary: searcher leak when openSearcher=false (was: searcher leak)
> searcher leak when openSearcher=false
> -------------------------------------
>
> Key: SOLR-3392
> URL: https://issues.apache.org/jira/browse/SOLR-3392
> Project: Solr
> Issue Type: Bug
> Reporter: Yonik Seeley
> Fix For: 4.0
>
>
> It appears we have a searcher leak where SolrIndexSearcher instances are sometimes not being closed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-3392) searcher leak
Posted by "Yonik Seeley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13258929#comment-13258929 ]
Yonik Seeley commented on SOLR-3392:
------------------------------------
bq. But they do? There were a number of failures when the number of closes/opens mismatched and failed the build.
Those normally don't fail on my box - just our Jenkins box which is really slow + has weird stuff like blackhole configured. Talking to Mark, it sounds like the cause of those open/close searcher mismatches are because another thread is still running (like a recovery/replication thread) and still has a searcher open (hence it doesn't look like a searcher management bug).
The type of bug I'm looking at would be a searcher management bug not related to thread safety and should be 100% reproducible.
> searcher leak
> -------------
>
> Key: SOLR-3392
> URL: https://issues.apache.org/jira/browse/SOLR-3392
> Project: Solr
> Issue Type: Bug
> Reporter: Yonik Seeley
>
> It appears we have a searcher leak where SolrIndexSearcher instances are sometimes not being closed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-3392) searcher leak when
openSearcher=false
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13259165#comment-13259165 ]
Robert Muir commented on SOLR-3392:
-----------------------------------
I think LuceneTestCase.testsFailed boolean variable should work?
LuceneTestCase.afterClass uses this logic too, for the same reason.
> searcher leak when openSearcher=false
> -------------------------------------
>
> Key: SOLR-3392
> URL: https://issues.apache.org/jira/browse/SOLR-3392
> Project: Solr
> Issue Type: Bug
> Reporter: Yonik Seeley
> Fix For: 4.0
>
>
> It appears we have a searcher leak where SolrIndexSearcher instances are sometimes not being closed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org