You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Thomas Connolly (JIRA)" <ji...@apache.org> on 2007/03/22 10:15:32 UTC
[jira] Created: (LUCENE-842) ParallelMultiSearcher memory leak
ParallelMultiSearcher memory leak
---------------------------------
Key: LUCENE-842
URL: https://issues.apache.org/jira/browse/LUCENE-842
Project: Lucene - Java
Issue Type: Bug
Components: Search
Affects Versions: 2.1
Environment: Windows XP SP1 and Red Hat EL 4
Reporter: Thomas Connolly
Priority: Critical
When using the org.apache.lucene.search.ParallelMultiSearcher to search on a single searcher (reading a single index), continuous runs result in a memory leak.
Substituting the MultiSearcher does not result in a memory leak. and is the workaround currently used.
And example of the code used is as follows. Note the close routine was added for the individual searchers and the MultiSearcher otherwise the was a leak in MultiSearcher.
private void doSearch(Search search)
{
IndexSearcher[] indexSearchers = null;
MultiSearcher multiSearcher = null;
try
{
indexSearchers = getIndexSearcher();
// aggregate the searches across multiple indexes
//multiSearcher = new ParallelMultiSearcher(indexSearchers);
multiSearcher = new MultiSearcher(indexSearchers);
final QueryParser parser = new QueryParser("content", new ExtendedStandardAnalyser());
final Query query = parser.parse(search.getQuery());
final Hits hits = multiSearcher.search(query, getFilter(search.getFilters()), getSort(search.getSort()));
// process hits...
}
finally
{
close(indexSearchers);
close(multiSearcher);
}
}
/**
* Close the index searchers.
*
* @param indexSearchers Index Searchers.
*/
private static void close(IndexSearcher[] indexSearchers)
{
if (indexSearchers != null)
{
for (IndexSearcher indexSearcher : indexSearchers)
{
try
{
indexSearcher.close();
}
catch (IOException ioex)
{
LOGGER.warn("Unable to close the index searcher!", ioex);
}
}
}
}
/**
* Close the multi-searcher.
*
* @param aMultiSearcher Index Searchers.
*/
private static void close(MultiSearcher aMultiSearcher)
{
try
{
aMultiSearcher.close();
}
catch (IOException ioex)
{
LOGGER.warn("Unable to close the multi searcher!", ioex);
}
}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-842) ParallelMultiSearcher memory leak
Posted by "Thomas Connolly (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12485445 ]
Thomas Connolly commented on LUCENE-842:
----------------------------------------
Just to add closure this issue had already been raised for Netbeans 5.5 and subsequently fixed.
See
http://www.netbeans.org/issues/show_bug.cgi?id=95020
Again thanks for the help resolving the profiler problem.
> ParallelMultiSearcher memory leak
> ---------------------------------
>
> Key: LUCENE-842
> URL: https://issues.apache.org/jira/browse/LUCENE-842
> Project: Lucene - Java
> Issue Type: Bug
> Components: Search
> Affects Versions: 2.1
> Environment: Windows XP SP2 and Red Hat EL 4
> Reporter: Thomas Connolly
> Priority: Critical
> Fix For: 2.1
>
> Attachments: search_test_gc.PNG, search_test_heap.PNG, TestParallelMultiSearcherMemLeak.java
>
>
> When using the org.apache.lucene.search.ParallelMultiSearcher to search on a single searcher (reading a single index), continuous runs result in a memory leak.
> Substituting the MultiSearcher does not result in a memory leak. and is the workaround currently used.
> And example of the code used is as follows. Note the close routine was added for the individual searchers and the MultiSearcher otherwise the was a leak in MultiSearcher.
> private void doSearch(Search search)
> {
> IndexSearcher[] indexSearchers = null;
>
> MultiSearcher multiSearcher = null;
> try
> {
> indexSearchers = getIndexSearcher();
>
> // aggregate the searches across multiple indexes
> multiSearcher = new ParallelMultiSearcher(indexSearchers); // causes LEAK BAD
> //multiSearcher = new MultiSearcher(indexSearchers); // NO leak GOOD
> final QueryParser parser = new QueryParser("content", new ExtendedStandardAnalyser());
> final Query query = parser.parse(search.getQuery());
>
> final Hits hits = multiSearcher.search(query, getFilter(search.getFilters()), getSort(search.getSort()));
> // process hits...
> }
> finally
> {
> close(indexSearchers);
> close(multiSearcher);
> }
> }
> /**
> * Close the index searchers.
> *
> * @param indexSearchers Index Searchers.
> */
> private static void close(IndexSearcher[] indexSearchers)
> {
> if (indexSearchers != null)
> {
> for (IndexSearcher indexSearcher : indexSearchers)
> {
> try
> {
> indexSearcher.close();
> }
> catch (IOException ioex)
> {
> LOGGER.warn("Unable to close the index searcher!", ioex);
> }
> }
> }
> }
>
> /**
> * Close the multi-searcher.
> *
> * @param aMultiSearcher Index Searchers.
> */
> private static void close(MultiSearcher aMultiSearcher)
> {
> try
> {
> aMultiSearcher.close();
> }
> catch (IOException ioex)
> {
> LOGGER.warn("Unable to close the multi searcher!", ioex);
> }
> }
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-842) ParallelMultiSearcher memory leak
Posted by "Thomas Connolly (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Connolly updated LUCENE-842:
-----------------------------------
Environment: Windows XP SP2 and Red Hat EL 4 (was: Windows XP SP1 and Red Hat EL 4)
> ParallelMultiSearcher memory leak
> ---------------------------------
>
> Key: LUCENE-842
> URL: https://issues.apache.org/jira/browse/LUCENE-842
> Project: Lucene - Java
> Issue Type: Bug
> Components: Search
> Affects Versions: 2.1
> Environment: Windows XP SP2 and Red Hat EL 4
> Reporter: Thomas Connolly
> Priority: Critical
>
> When using the org.apache.lucene.search.ParallelMultiSearcher to search on a single searcher (reading a single index), continuous runs result in a memory leak.
> Substituting the MultiSearcher does not result in a memory leak. and is the workaround currently used.
> And example of the code used is as follows. Note the close routine was added for the individual searchers and the MultiSearcher otherwise the was a leak in MultiSearcher.
> private void doSearch(Search search)
> {
> IndexSearcher[] indexSearchers = null;
>
> MultiSearcher multiSearcher = null;
> try
> {
> indexSearchers = getIndexSearcher();
>
> // aggregate the searches across multiple indexes
> //multiSearcher = new ParallelMultiSearcher(indexSearchers);
> multiSearcher = new MultiSearcher(indexSearchers);
> final QueryParser parser = new QueryParser("content", new ExtendedStandardAnalyser());
> final Query query = parser.parse(search.getQuery());
>
> final Hits hits = multiSearcher.search(query, getFilter(search.getFilters()), getSort(search.getSort()));
> // process hits...
> }
> finally
> {
> close(indexSearchers);
> close(multiSearcher);
> }
> }
> /**
> * Close the index searchers.
> *
> * @param indexSearchers Index Searchers.
> */
> private static void close(IndexSearcher[] indexSearchers)
> {
> if (indexSearchers != null)
> {
> for (IndexSearcher indexSearcher : indexSearchers)
> {
> try
> {
> indexSearcher.close();
> }
> catch (IOException ioex)
> {
> LOGGER.warn("Unable to close the index searcher!", ioex);
> }
> }
> }
> }
>
> /**
> * Close the multi-searcher.
> *
> * @param aMultiSearcher Index Searchers.
> */
> private static void close(MultiSearcher aMultiSearcher)
> {
> try
> {
> aMultiSearcher.close();
> }
> catch (IOException ioex)
> {
> LOGGER.warn("Unable to close the multi searcher!", ioex);
> }
> }
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Closed: (LUCENE-842) ParallelMultiSearcher memory leak
Posted by "Thomas Connolly (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Connolly closed LUCENE-842.
----------------------------------
Resolution: Invalid
Fix Version/s: 2.1
It was not a bug with lucene rather the netbeans 5.5 memory profiler was providing misleading information. I've attached two screenshots that show the heap and gc using the test case provided.
There may be an issue with netbeans around the profiling of joined threads, regardless it is not a bug in lucene.
I do need to open and close a searchers, in each call. Basically the index(s) can be changed based upon the other events.
Please close the issue. Thank you for answering so quickly. It is a great piece of software.
> ParallelMultiSearcher memory leak
> ---------------------------------
>
> Key: LUCENE-842
> URL: https://issues.apache.org/jira/browse/LUCENE-842
> Project: Lucene - Java
> Issue Type: Bug
> Components: Search
> Affects Versions: 2.1
> Environment: Windows XP SP2 and Red Hat EL 4
> Reporter: Thomas Connolly
> Priority: Critical
> Fix For: 2.1
>
> Attachments: search_test_gc.PNG, search_test_heap.PNG, TestParallelMultiSearcherMemLeak.java
>
>
> When using the org.apache.lucene.search.ParallelMultiSearcher to search on a single searcher (reading a single index), continuous runs result in a memory leak.
> Substituting the MultiSearcher does not result in a memory leak. and is the workaround currently used.
> And example of the code used is as follows. Note the close routine was added for the individual searchers and the MultiSearcher otherwise the was a leak in MultiSearcher.
> private void doSearch(Search search)
> {
> IndexSearcher[] indexSearchers = null;
>
> MultiSearcher multiSearcher = null;
> try
> {
> indexSearchers = getIndexSearcher();
>
> // aggregate the searches across multiple indexes
> multiSearcher = new ParallelMultiSearcher(indexSearchers); // causes LEAK BAD
> //multiSearcher = new MultiSearcher(indexSearchers); // NO leak GOOD
> final QueryParser parser = new QueryParser("content", new ExtendedStandardAnalyser());
> final Query query = parser.parse(search.getQuery());
>
> final Hits hits = multiSearcher.search(query, getFilter(search.getFilters()), getSort(search.getSort()));
> // process hits...
> }
> finally
> {
> close(indexSearchers);
> close(multiSearcher);
> }
> }
> /**
> * Close the index searchers.
> *
> * @param indexSearchers Index Searchers.
> */
> private static void close(IndexSearcher[] indexSearchers)
> {
> if (indexSearchers != null)
> {
> for (IndexSearcher indexSearcher : indexSearchers)
> {
> try
> {
> indexSearcher.close();
> }
> catch (IOException ioex)
> {
> LOGGER.warn("Unable to close the index searcher!", ioex);
> }
> }
> }
> }
>
> /**
> * Close the multi-searcher.
> *
> * @param aMultiSearcher Index Searchers.
> */
> private static void close(MultiSearcher aMultiSearcher)
> {
> try
> {
> aMultiSearcher.close();
> }
> catch (IOException ioex)
> {
> LOGGER.warn("Unable to close the multi searcher!", ioex);
> }
> }
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-842) ParallelMultiSearcher memory leak
Posted by "Otis Gospodnetic (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12483314 ]
Otis Gospodnetic commented on LUCENE-842:
-----------------------------------------
Thomas:
Can you write a JUnit test that builds a sample index and then runs PMS over it, eventually OOM-ing?
>From the cursory glance over your code, it looks like you are closing your searchers after every single search. If that is indeed what you are doing, this may be the source of the memory leak - try reusing your searchers.
Also, why create a new (P)MS on every search? Create it once, and reuse it.
I'll leave this issue open for a few more days, but from the above code, it just looks like misuse of the Lucene API.
> ParallelMultiSearcher memory leak
> ---------------------------------
>
> Key: LUCENE-842
> URL: https://issues.apache.org/jira/browse/LUCENE-842
> Project: Lucene - Java
> Issue Type: Bug
> Components: Search
> Affects Versions: 2.1
> Environment: Windows XP SP2 and Red Hat EL 4
> Reporter: Thomas Connolly
> Priority: Critical
> Attachments: TestParallelMultiSearcherMemLeak.java
>
>
> When using the org.apache.lucene.search.ParallelMultiSearcher to search on a single searcher (reading a single index), continuous runs result in a memory leak.
> Substituting the MultiSearcher does not result in a memory leak. and is the workaround currently used.
> And example of the code used is as follows. Note the close routine was added for the individual searchers and the MultiSearcher otherwise the was a leak in MultiSearcher.
> private void doSearch(Search search)
> {
> IndexSearcher[] indexSearchers = null;
>
> MultiSearcher multiSearcher = null;
> try
> {
> indexSearchers = getIndexSearcher();
>
> // aggregate the searches across multiple indexes
> multiSearcher = new ParallelMultiSearcher(indexSearchers); // causes LEAK BAD
> //multiSearcher = new MultiSearcher(indexSearchers); // NO leak GOOD
> final QueryParser parser = new QueryParser("content", new ExtendedStandardAnalyser());
> final Query query = parser.parse(search.getQuery());
>
> final Hits hits = multiSearcher.search(query, getFilter(search.getFilters()), getSort(search.getSort()));
> // process hits...
> }
> finally
> {
> close(indexSearchers);
> close(multiSearcher);
> }
> }
> /**
> * Close the index searchers.
> *
> * @param indexSearchers Index Searchers.
> */
> private static void close(IndexSearcher[] indexSearchers)
> {
> if (indexSearchers != null)
> {
> for (IndexSearcher indexSearcher : indexSearchers)
> {
> try
> {
> indexSearcher.close();
> }
> catch (IOException ioex)
> {
> LOGGER.warn("Unable to close the index searcher!", ioex);
> }
> }
> }
> }
>
> /**
> * Close the multi-searcher.
> *
> * @param aMultiSearcher Index Searchers.
> */
> private static void close(MultiSearcher aMultiSearcher)
> {
> try
> {
> aMultiSearcher.close();
> }
> catch (IOException ioex)
> {
> LOGGER.warn("Unable to close the multi searcher!", ioex);
> }
> }
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-842) ParallelMultiSearcher memory leak
Posted by "Thomas Connolly (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Connolly updated LUCENE-842:
-----------------------------------
Attachment: search_test_gc.PNG
Netbeans 5.5 garbage collection stats
> ParallelMultiSearcher memory leak
> ---------------------------------
>
> Key: LUCENE-842
> URL: https://issues.apache.org/jira/browse/LUCENE-842
> Project: Lucene - Java
> Issue Type: Bug
> Components: Search
> Affects Versions: 2.1
> Environment: Windows XP SP2 and Red Hat EL 4
> Reporter: Thomas Connolly
> Priority: Critical
> Attachments: search_test_gc.PNG, search_test_heap.PNG, TestParallelMultiSearcherMemLeak.java
>
>
> When using the org.apache.lucene.search.ParallelMultiSearcher to search on a single searcher (reading a single index), continuous runs result in a memory leak.
> Substituting the MultiSearcher does not result in a memory leak. and is the workaround currently used.
> And example of the code used is as follows. Note the close routine was added for the individual searchers and the MultiSearcher otherwise the was a leak in MultiSearcher.
> private void doSearch(Search search)
> {
> IndexSearcher[] indexSearchers = null;
>
> MultiSearcher multiSearcher = null;
> try
> {
> indexSearchers = getIndexSearcher();
>
> // aggregate the searches across multiple indexes
> multiSearcher = new ParallelMultiSearcher(indexSearchers); // causes LEAK BAD
> //multiSearcher = new MultiSearcher(indexSearchers); // NO leak GOOD
> final QueryParser parser = new QueryParser("content", new ExtendedStandardAnalyser());
> final Query query = parser.parse(search.getQuery());
>
> final Hits hits = multiSearcher.search(query, getFilter(search.getFilters()), getSort(search.getSort()));
> // process hits...
> }
> finally
> {
> close(indexSearchers);
> close(multiSearcher);
> }
> }
> /**
> * Close the index searchers.
> *
> * @param indexSearchers Index Searchers.
> */
> private static void close(IndexSearcher[] indexSearchers)
> {
> if (indexSearchers != null)
> {
> for (IndexSearcher indexSearcher : indexSearchers)
> {
> try
> {
> indexSearcher.close();
> }
> catch (IOException ioex)
> {
> LOGGER.warn("Unable to close the index searcher!", ioex);
> }
> }
> }
> }
>
> /**
> * Close the multi-searcher.
> *
> * @param aMultiSearcher Index Searchers.
> */
> private static void close(MultiSearcher aMultiSearcher)
> {
> try
> {
> aMultiSearcher.close();
> }
> catch (IOException ioex)
> {
> LOGGER.warn("Unable to close the multi searcher!", ioex);
> }
> }
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-842) ParallelMultiSearcher memory leak
Posted by "Doron Cohen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doron Cohen updated LUCENE-842:
-------------------------------
Attachment: TestParallelMultiSearcherMemLeak.java
Best always is a unit test that fails due to the bug. This also allows to verify later that a fix actually fixes the bug.
I packed the sample code in a Junit test (attached).
It passes for me.
Thomas, Does it fail for you?
If not, could you modify it to demonstrate the bug?
Thanks,
Doron
> ParallelMultiSearcher memory leak
> ---------------------------------
>
> Key: LUCENE-842
> URL: https://issues.apache.org/jira/browse/LUCENE-842
> Project: Lucene - Java
> Issue Type: Bug
> Components: Search
> Affects Versions: 2.1
> Environment: Windows XP SP2 and Red Hat EL 4
> Reporter: Thomas Connolly
> Priority: Critical
> Attachments: TestParallelMultiSearcherMemLeak.java
>
>
> When using the org.apache.lucene.search.ParallelMultiSearcher to search on a single searcher (reading a single index), continuous runs result in a memory leak.
> Substituting the MultiSearcher does not result in a memory leak. and is the workaround currently used.
> And example of the code used is as follows. Note the close routine was added for the individual searchers and the MultiSearcher otherwise the was a leak in MultiSearcher.
> private void doSearch(Search search)
> {
> IndexSearcher[] indexSearchers = null;
>
> MultiSearcher multiSearcher = null;
> try
> {
> indexSearchers = getIndexSearcher();
>
> // aggregate the searches across multiple indexes
> multiSearcher = new ParallelMultiSearcher(indexSearchers); // causes LEAK BAD
> //multiSearcher = new MultiSearcher(indexSearchers); // NO leak GOOD
> final QueryParser parser = new QueryParser("content", new ExtendedStandardAnalyser());
> final Query query = parser.parse(search.getQuery());
>
> final Hits hits = multiSearcher.search(query, getFilter(search.getFilters()), getSort(search.getSort()));
> // process hits...
> }
> finally
> {
> close(indexSearchers);
> close(multiSearcher);
> }
> }
> /**
> * Close the index searchers.
> *
> * @param indexSearchers Index Searchers.
> */
> private static void close(IndexSearcher[] indexSearchers)
> {
> if (indexSearchers != null)
> {
> for (IndexSearcher indexSearcher : indexSearchers)
> {
> try
> {
> indexSearcher.close();
> }
> catch (IOException ioex)
> {
> LOGGER.warn("Unable to close the index searcher!", ioex);
> }
> }
> }
> }
>
> /**
> * Close the multi-searcher.
> *
> * @param aMultiSearcher Index Searchers.
> */
> private static void close(MultiSearcher aMultiSearcher)
> {
> try
> {
> aMultiSearcher.close();
> }
> catch (IOException ioex)
> {
> LOGGER.warn("Unable to close the multi searcher!", ioex);
> }
> }
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-842) ParallelMultiSearcher memory leak
Posted by "Thomas Connolly (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Connolly updated LUCENE-842:
-----------------------------------
Attachment: search_test_heap.PNG
Netbeans 5.5 profile of heap
> ParallelMultiSearcher memory leak
> ---------------------------------
>
> Key: LUCENE-842
> URL: https://issues.apache.org/jira/browse/LUCENE-842
> Project: Lucene - Java
> Issue Type: Bug
> Components: Search
> Affects Versions: 2.1
> Environment: Windows XP SP2 and Red Hat EL 4
> Reporter: Thomas Connolly
> Priority: Critical
> Attachments: search_test_heap.PNG, TestParallelMultiSearcherMemLeak.java
>
>
> When using the org.apache.lucene.search.ParallelMultiSearcher to search on a single searcher (reading a single index), continuous runs result in a memory leak.
> Substituting the MultiSearcher does not result in a memory leak. and is the workaround currently used.
> And example of the code used is as follows. Note the close routine was added for the individual searchers and the MultiSearcher otherwise the was a leak in MultiSearcher.
> private void doSearch(Search search)
> {
> IndexSearcher[] indexSearchers = null;
>
> MultiSearcher multiSearcher = null;
> try
> {
> indexSearchers = getIndexSearcher();
>
> // aggregate the searches across multiple indexes
> multiSearcher = new ParallelMultiSearcher(indexSearchers); // causes LEAK BAD
> //multiSearcher = new MultiSearcher(indexSearchers); // NO leak GOOD
> final QueryParser parser = new QueryParser("content", new ExtendedStandardAnalyser());
> final Query query = parser.parse(search.getQuery());
>
> final Hits hits = multiSearcher.search(query, getFilter(search.getFilters()), getSort(search.getSort()));
> // process hits...
> }
> finally
> {
> close(indexSearchers);
> close(multiSearcher);
> }
> }
> /**
> * Close the index searchers.
> *
> * @param indexSearchers Index Searchers.
> */
> private static void close(IndexSearcher[] indexSearchers)
> {
> if (indexSearchers != null)
> {
> for (IndexSearcher indexSearcher : indexSearchers)
> {
> try
> {
> indexSearcher.close();
> }
> catch (IOException ioex)
> {
> LOGGER.warn("Unable to close the index searcher!", ioex);
> }
> }
> }
> }
>
> /**
> * Close the multi-searcher.
> *
> * @param aMultiSearcher Index Searchers.
> */
> private static void close(MultiSearcher aMultiSearcher)
> {
> try
> {
> aMultiSearcher.close();
> }
> catch (IOException ioex)
> {
> LOGGER.warn("Unable to close the multi searcher!", ioex);
> }
> }
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-842) ParallelMultiSearcher memory leak
Posted by "Doron Cohen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12483264 ]
Doron Cohen commented on LUCENE-842:
------------------------------------
It is not clear to me how this code demonstrates a mem leak. The program itself is not very useful for showing/fixing that because it is not runnable as is - lots of getXXYZ() methods with unknown logic - others cannot compile and run it.
What I get from the problem description and code is this:
<quote>
If someone repeats the following sequence of steps over and over again:
{
[1] open one index searcher S over an index;
[2] create a parallel multi searcher P over S;
[3] search using P;
[4] close P;
[5] close S;
}
then eventually a memory leak will be evident.
</quote>
And it seems you think that a single searcher is the case that causes the leak. Do you mean that with (say) two searchers there want be a leak?
> ParallelMultiSearcher memory leak
> ---------------------------------
>
> Key: LUCENE-842
> URL: https://issues.apache.org/jira/browse/LUCENE-842
> Project: Lucene - Java
> Issue Type: Bug
> Components: Search
> Affects Versions: 2.1
> Environment: Windows XP SP2 and Red Hat EL 4
> Reporter: Thomas Connolly
> Priority: Critical
>
> When using the org.apache.lucene.search.ParallelMultiSearcher to search on a single searcher (reading a single index), continuous runs result in a memory leak.
> Substituting the MultiSearcher does not result in a memory leak. and is the workaround currently used.
> And example of the code used is as follows. Note the close routine was added for the individual searchers and the MultiSearcher otherwise the was a leak in MultiSearcher.
> private void doSearch(Search search)
> {
> IndexSearcher[] indexSearchers = null;
>
> MultiSearcher multiSearcher = null;
> try
> {
> indexSearchers = getIndexSearcher();
>
> // aggregate the searches across multiple indexes
> //multiSearcher = new ParallelMultiSearcher(indexSearchers);
> multiSearcher = new MultiSearcher(indexSearchers);
> final QueryParser parser = new QueryParser("content", new ExtendedStandardAnalyser());
> final Query query = parser.parse(search.getQuery());
>
> final Hits hits = multiSearcher.search(query, getFilter(search.getFilters()), getSort(search.getSort()));
> // process hits...
> }
> finally
> {
> close(indexSearchers);
> close(multiSearcher);
> }
> }
> /**
> * Close the index searchers.
> *
> * @param indexSearchers Index Searchers.
> */
> private static void close(IndexSearcher[] indexSearchers)
> {
> if (indexSearchers != null)
> {
> for (IndexSearcher indexSearcher : indexSearchers)
> {
> try
> {
> indexSearcher.close();
> }
> catch (IOException ioex)
> {
> LOGGER.warn("Unable to close the index searcher!", ioex);
> }
> }
> }
> }
>
> /**
> * Close the multi-searcher.
> *
> * @param aMultiSearcher Index Searchers.
> */
> private static void close(MultiSearcher aMultiSearcher)
> {
> try
> {
> aMultiSearcher.close();
> }
> catch (IOException ioex)
> {
> LOGGER.warn("Unable to close the multi searcher!", ioex);
> }
> }
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-842) ParallelMultiSearcher memory leak
Posted by "Doron Cohen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12483418 ]
Doron Cohen commented on LUCENE-842:
------------------------------------
Thomas, thanks for following up closely on this (and great that this is not a real bug :-) )
Doron
> ParallelMultiSearcher memory leak
> ---------------------------------
>
> Key: LUCENE-842
> URL: https://issues.apache.org/jira/browse/LUCENE-842
> Project: Lucene - Java
> Issue Type: Bug
> Components: Search
> Affects Versions: 2.1
> Environment: Windows XP SP2 and Red Hat EL 4
> Reporter: Thomas Connolly
> Priority: Critical
> Fix For: 2.1
>
> Attachments: search_test_gc.PNG, search_test_heap.PNG, TestParallelMultiSearcherMemLeak.java
>
>
> When using the org.apache.lucene.search.ParallelMultiSearcher to search on a single searcher (reading a single index), continuous runs result in a memory leak.
> Substituting the MultiSearcher does not result in a memory leak. and is the workaround currently used.
> And example of the code used is as follows. Note the close routine was added for the individual searchers and the MultiSearcher otherwise the was a leak in MultiSearcher.
> private void doSearch(Search search)
> {
> IndexSearcher[] indexSearchers = null;
>
> MultiSearcher multiSearcher = null;
> try
> {
> indexSearchers = getIndexSearcher();
>
> // aggregate the searches across multiple indexes
> multiSearcher = new ParallelMultiSearcher(indexSearchers); // causes LEAK BAD
> //multiSearcher = new MultiSearcher(indexSearchers); // NO leak GOOD
> final QueryParser parser = new QueryParser("content", new ExtendedStandardAnalyser());
> final Query query = parser.parse(search.getQuery());
>
> final Hits hits = multiSearcher.search(query, getFilter(search.getFilters()), getSort(search.getSort()));
> // process hits...
> }
> finally
> {
> close(indexSearchers);
> close(multiSearcher);
> }
> }
> /**
> * Close the index searchers.
> *
> * @param indexSearchers Index Searchers.
> */
> private static void close(IndexSearcher[] indexSearchers)
> {
> if (indexSearchers != null)
> {
> for (IndexSearcher indexSearcher : indexSearchers)
> {
> try
> {
> indexSearcher.close();
> }
> catch (IOException ioex)
> {
> LOGGER.warn("Unable to close the index searcher!", ioex);
> }
> }
> }
> }
>
> /**
> * Close the multi-searcher.
> *
> * @param aMultiSearcher Index Searchers.
> */
> private static void close(MultiSearcher aMultiSearcher)
> {
> try
> {
> aMultiSearcher.close();
> }
> catch (IOException ioex)
> {
> LOGGER.warn("Unable to close the multi searcher!", ioex);
> }
> }
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-842) ParallelMultiSearcher memory leak
Posted by "Thomas Connolly (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Connolly updated LUCENE-842:
-----------------------------------
Description:
When using the org.apache.lucene.search.ParallelMultiSearcher to search on a single searcher (reading a single index), continuous runs result in a memory leak.
Substituting the MultiSearcher does not result in a memory leak. and is the workaround currently used.
And example of the code used is as follows. Note the close routine was added for the individual searchers and the MultiSearcher otherwise the was a leak in MultiSearcher.
private void doSearch(Search search)
{
IndexSearcher[] indexSearchers = null;
MultiSearcher multiSearcher = null;
try
{
indexSearchers = getIndexSearcher();
// aggregate the searches across multiple indexes
multiSearcher = new ParallelMultiSearcher(indexSearchers); // causes LEAK BAD
//multiSearcher = new MultiSearcher(indexSearchers); // NO leak GOOD
final QueryParser parser = new QueryParser("content", new ExtendedStandardAnalyser());
final Query query = parser.parse(search.getQuery());
final Hits hits = multiSearcher.search(query, getFilter(search.getFilters()), getSort(search.getSort()));
// process hits...
}
finally
{
close(indexSearchers);
close(multiSearcher);
}
}
/**
* Close the index searchers.
*
* @param indexSearchers Index Searchers.
*/
private static void close(IndexSearcher[] indexSearchers)
{
if (indexSearchers != null)
{
for (IndexSearcher indexSearcher : indexSearchers)
{
try
{
indexSearcher.close();
}
catch (IOException ioex)
{
LOGGER.warn("Unable to close the index searcher!", ioex);
}
}
}
}
/**
* Close the multi-searcher.
*
* @param aMultiSearcher Index Searchers.
*/
private static void close(MultiSearcher aMultiSearcher)
{
try
{
aMultiSearcher.close();
}
catch (IOException ioex)
{
LOGGER.warn("Unable to close the multi searcher!", ioex);
}
}
was:
When using the org.apache.lucene.search.ParallelMultiSearcher to search on a single searcher (reading a single index), continuous runs result in a memory leak.
Substituting the MultiSearcher does not result in a memory leak. and is the workaround currently used.
And example of the code used is as follows. Note the close routine was added for the individual searchers and the MultiSearcher otherwise the was a leak in MultiSearcher.
private void doSearch(Search search)
{
IndexSearcher[] indexSearchers = null;
MultiSearcher multiSearcher = null;
try
{
indexSearchers = getIndexSearcher();
// aggregate the searches across multiple indexes
//multiSearcher = new ParallelMultiSearcher(indexSearchers);
multiSearcher = new MultiSearcher(indexSearchers);
final QueryParser parser = new QueryParser("content", new ExtendedStandardAnalyser());
final Query query = parser.parse(search.getQuery());
final Hits hits = multiSearcher.search(query, getFilter(search.getFilters()), getSort(search.getSort()));
// process hits...
}
finally
{
close(indexSearchers);
close(multiSearcher);
}
}
/**
* Close the index searchers.
*
* @param indexSearchers Index Searchers.
*/
private static void close(IndexSearcher[] indexSearchers)
{
if (indexSearchers != null)
{
for (IndexSearcher indexSearcher : indexSearchers)
{
try
{
indexSearcher.close();
}
catch (IOException ioex)
{
LOGGER.warn("Unable to close the index searcher!", ioex);
}
}
}
}
/**
* Close the multi-searcher.
*
* @param aMultiSearcher Index Searchers.
*/
private static void close(MultiSearcher aMultiSearcher)
{
try
{
aMultiSearcher.close();
}
catch (IOException ioex)
{
LOGGER.warn("Unable to close the multi searcher!", ioex);
}
}
Updated the code example to leak.
Doron what do you suggest? Do you want me to attached the search index, libraries, and a main routine? It is pretty easy to execute this code, I'm simply showing the simple logic.
The first close, on the searchers, should not be necessary as the MultiSearcher should close all searchers, but I found it is.
> ParallelMultiSearcher memory leak
> ---------------------------------
>
> Key: LUCENE-842
> URL: https://issues.apache.org/jira/browse/LUCENE-842
> Project: Lucene - Java
> Issue Type: Bug
> Components: Search
> Affects Versions: 2.1
> Environment: Windows XP SP2 and Red Hat EL 4
> Reporter: Thomas Connolly
> Priority: Critical
>
> When using the org.apache.lucene.search.ParallelMultiSearcher to search on a single searcher (reading a single index), continuous runs result in a memory leak.
> Substituting the MultiSearcher does not result in a memory leak. and is the workaround currently used.
> And example of the code used is as follows. Note the close routine was added for the individual searchers and the MultiSearcher otherwise the was a leak in MultiSearcher.
> private void doSearch(Search search)
> {
> IndexSearcher[] indexSearchers = null;
>
> MultiSearcher multiSearcher = null;
> try
> {
> indexSearchers = getIndexSearcher();
>
> // aggregate the searches across multiple indexes
> multiSearcher = new ParallelMultiSearcher(indexSearchers); // causes LEAK BAD
> //multiSearcher = new MultiSearcher(indexSearchers); // NO leak GOOD
> final QueryParser parser = new QueryParser("content", new ExtendedStandardAnalyser());
> final Query query = parser.parse(search.getQuery());
>
> final Hits hits = multiSearcher.search(query, getFilter(search.getFilters()), getSort(search.getSort()));
> // process hits...
> }
> finally
> {
> close(indexSearchers);
> close(multiSearcher);
> }
> }
> /**
> * Close the index searchers.
> *
> * @param indexSearchers Index Searchers.
> */
> private static void close(IndexSearcher[] indexSearchers)
> {
> if (indexSearchers != null)
> {
> for (IndexSearcher indexSearcher : indexSearchers)
> {
> try
> {
> indexSearcher.close();
> }
> catch (IOException ioex)
> {
> LOGGER.warn("Unable to close the index searcher!", ioex);
> }
> }
> }
> }
>
> /**
> * Close the multi-searcher.
> *
> * @param aMultiSearcher Index Searchers.
> */
> private static void close(MultiSearcher aMultiSearcher)
> {
> try
> {
> aMultiSearcher.close();
> }
> catch (IOException ioex)
> {
> LOGGER.warn("Unable to close the multi searcher!", ioex);
> }
> }
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org