You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-dev@lucene.apache.org by "Marc Sturlese (JIRA)" <ji...@apache.org> on 2009/03/16 19:51:50 UTC
[jira] Created: (SOLR-1070) IndexSearcher not closed after indexing
with DataImportHandler
IndexSearcher not closed after indexing with DataImportHandler
--------------------------------------------------------------
Key: SOLR-1070
URL: https://issues.apache.org/jira/browse/SOLR-1070
Project: Solr
Issue Type: Bug
Components: clients - java
Affects Versions: 1.4
Environment: Debian 2.6.26-1-xen-amd64
Tomcat 5.5
Reporter: Marc Sturlese
Fix For: 1.4
What I have noticed is afecting to the nightly build using multicore. I am using tomcat 5.5 server and Debian 2.6.26-1-xen-amd64.
After indexing and optimizing with DataImporthandler the old IndexSearcher is not being proper closed.
To reproduce the bug I have done a full-import from a mysql database optimizing at the end. Once done I change the data from the database and index optimizing again.
When now I do a lsof | grep solr I can see that tomcat is holding the deleted files from old index.
I can see the error in the stats screen aswell... if I do 4 full-imports, stats will show me 3 opened IndexSearchers.
As I index on the master and never use it to serve search requests I have sorted it modifying the function getSearcher in SolrCore.java
There's a part in the source where is coded:
// we are all done with the old searcher we used
// for warming...
if (currSearcherHolderF!=null){currSearcherHolderF.decref();}
I force there to close the indexSearcher and remvoe it from the RefCounted:
// we are all done with the old searcher we used
// for warming...
if (currSearcherHolderF!=null){
currSearcherHolderF.get().close();
currSearcherHolderF.decref();
_searchers.remove();
}
As Yonik told me:
>> Forcing it to close at the point you did is unsafe since other threads
>> may still be using that searcher.
My solution is definitely not good if search requests must be served on that core of that server...
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Closed: (SOLR-1070) IndexSearcher not closed after indexing
with DataImportHandler
Posted by "Marc Sturlese (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-1070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Marc Sturlese closed SOLR-1070.
-------------------------------
Resolution: Fixed
It was actually not a bug, just a bad use of the Context of a Transformer of DataImportHandler.
> IndexSearcher not closed after indexing with DataImportHandler
> --------------------------------------------------------------
>
> Key: SOLR-1070
> URL: https://issues.apache.org/jira/browse/SOLR-1070
> Project: Solr
> Issue Type: Bug
> Components: clients - java
> Affects Versions: 1.4
> Environment: Debian 2.6.26-1-xen-amd64
> Tomcat 5.5
> Reporter: Marc Sturlese
> Fix For: 1.4
>
>
> What I have noticed is afecting to the nightly build using multicore. I am using tomcat 5.5 server and Debian 2.6.26-1-xen-amd64.
> After indexing and optimizing with DataImporthandler the old IndexSearcher is not being proper closed.
> To reproduce the bug I have done a full-import from a mysql database optimizing at the end. Once done I change the data from the database and index optimizing again.
> When now I do a lsof | grep solr I can see that tomcat is holding the deleted files from old index.
> I can see the error in the stats screen aswell... if I do 4 full-imports, stats will show me 3 opened IndexSearchers.
> As I index on the master and never use it to serve search requests I have sorted it modifying the function getSearcher in SolrCore.java
> There's a part in the source where is coded:
> // we are all done with the old searcher we used
> // for warming...
> if (currSearcherHolderF!=null){currSearcherHolderF.decref();}
> I force there to close the indexSearcher and remvoe it from the RefCounted:
> // we are all done with the old searcher we used
> // for warming...
> if (currSearcherHolderF!=null){
> currSearcherHolderF.get().close();
> currSearcherHolderF.decref();
> _searchers.remove();
> }
> As Yonik told me:
> >> Forcing it to close at the point you did is unsafe since other threads
> >> may still be using that searcher.
> My solution is definitely not good if search requests must be served on that core of that server...
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-1070) IndexSearcher not closed after
indexing with DataImportHandler
Posted by "Shalin Shekhar Mangar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-1070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12683131#action_12683131 ]
Shalin Shekhar Mangar commented on SOLR-1070:
---------------------------------------------
I don't think this is related to DataImportHandler. DIH calls commit using a CommitUpdateCommand the same way as an xml commit request would.
> IndexSearcher not closed after indexing with DataImportHandler
> --------------------------------------------------------------
>
> Key: SOLR-1070
> URL: https://issues.apache.org/jira/browse/SOLR-1070
> Project: Solr
> Issue Type: Bug
> Components: clients - java
> Affects Versions: 1.4
> Environment: Debian 2.6.26-1-xen-amd64
> Tomcat 5.5
> Reporter: Marc Sturlese
> Fix For: 1.4
>
>
> What I have noticed is afecting to the nightly build using multicore. I am using tomcat 5.5 server and Debian 2.6.26-1-xen-amd64.
> After indexing and optimizing with DataImporthandler the old IndexSearcher is not being proper closed.
> To reproduce the bug I have done a full-import from a mysql database optimizing at the end. Once done I change the data from the database and index optimizing again.
> When now I do a lsof | grep solr I can see that tomcat is holding the deleted files from old index.
> I can see the error in the stats screen aswell... if I do 4 full-imports, stats will show me 3 opened IndexSearchers.
> As I index on the master and never use it to serve search requests I have sorted it modifying the function getSearcher in SolrCore.java
> There's a part in the source where is coded:
> // we are all done with the old searcher we used
> // for warming...
> if (currSearcherHolderF!=null){currSearcherHolderF.decref();}
> I force there to close the indexSearcher and remvoe it from the RefCounted:
> // we are all done with the old searcher we used
> // for warming...
> if (currSearcherHolderF!=null){
> currSearcherHolderF.get().close();
> currSearcherHolderF.decref();
> _searchers.remove();
> }
> As Yonik told me:
> >> Forcing it to close at the point you did is unsafe since other threads
> >> may still be using that searcher.
> My solution is definitely not good if search requests must be served on that core of that server...
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-1070) IndexSearcher not closed after
indexing with DataImportHandler
Posted by "Noble Paul (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-1070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12683314#action_12683314 ]
Noble Paul commented on SOLR-1070:
----------------------------------
this is the code which does a commit in DIH
{code:java}
//SolrWriter#commit()
CommitUpdateCommand commit = new CommitUpdateCommand(optimize);
updateProcessor.processCommit(commit);
{code}
is there anything else we need to do to fix this
> IndexSearcher not closed after indexing with DataImportHandler
> --------------------------------------------------------------
>
> Key: SOLR-1070
> URL: https://issues.apache.org/jira/browse/SOLR-1070
> Project: Solr
> Issue Type: Bug
> Components: clients - java
> Affects Versions: 1.4
> Environment: Debian 2.6.26-1-xen-amd64
> Tomcat 5.5
> Reporter: Marc Sturlese
> Fix For: 1.4
>
>
> What I have noticed is afecting to the nightly build using multicore. I am using tomcat 5.5 server and Debian 2.6.26-1-xen-amd64.
> After indexing and optimizing with DataImporthandler the old IndexSearcher is not being proper closed.
> To reproduce the bug I have done a full-import from a mysql database optimizing at the end. Once done I change the data from the database and index optimizing again.
> When now I do a lsof | grep solr I can see that tomcat is holding the deleted files from old index.
> I can see the error in the stats screen aswell... if I do 4 full-imports, stats will show me 3 opened IndexSearchers.
> As I index on the master and never use it to serve search requests I have sorted it modifying the function getSearcher in SolrCore.java
> There's a part in the source where is coded:
> // we are all done with the old searcher we used
> // for warming...
> if (currSearcherHolderF!=null){currSearcherHolderF.decref();}
> I force there to close the indexSearcher and remvoe it from the RefCounted:
> // we are all done with the old searcher we used
> // for warming...
> if (currSearcherHolderF!=null){
> currSearcherHolderF.get().close();
> currSearcherHolderF.decref();
> _searchers.remove();
> }
> As Yonik told me:
> >> Forcing it to close at the point you did is unsafe since other threads
> >> may still be using that searcher.
> My solution is definitely not good if search requests must be served on that core of that server...
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-1070) IndexSearcher not closed after indexing
with DataImportHandler
Posted by "Marc Sturlese (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-1070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Marc Sturlese updated SOLR-1070:
--------------------------------
Comment: was deleted
(was: It was actually not a bug, just a bad use of the Context of a Transformer of DataImportHandler.)
> IndexSearcher not closed after indexing with DataImportHandler
> --------------------------------------------------------------
>
> Key: SOLR-1070
> URL: https://issues.apache.org/jira/browse/SOLR-1070
> Project: Solr
> Issue Type: Bug
> Components: clients - java
> Affects Versions: 1.4
> Environment: Debian 2.6.26-1-xen-amd64
> Tomcat 5.5
> Reporter: Marc Sturlese
> Fix For: 1.4
>
>
> What I have noticed is afecting to the nightly build using multicore. I am using tomcat 5.5 server and Debian 2.6.26-1-xen-amd64.
> After indexing and optimizing with DataImporthandler the old IndexSearcher is not being proper closed.
> To reproduce the bug I have done a full-import from a mysql database optimizing at the end. Once done I change the data from the database and index optimizing again.
> When now I do a lsof | grep solr I can see that tomcat is holding the deleted files from old index.
> I can see the error in the stats screen aswell... if I do 4 full-imports, stats will show me 3 opened IndexSearchers.
> As I index on the master and never use it to serve search requests I have sorted it modifying the function getSearcher in SolrCore.java
> There's a part in the source where is coded:
> // we are all done with the old searcher we used
> // for warming...
> if (currSearcherHolderF!=null){currSearcherHolderF.decref();}
> I force there to close the indexSearcher and remvoe it from the RefCounted:
> // we are all done with the old searcher we used
> // for warming...
> if (currSearcherHolderF!=null){
> currSearcherHolderF.get().close();
> currSearcherHolderF.decref();
> _searchers.remove();
> }
> As Yonik told me:
> >> Forcing it to close at the point you did is unsafe since other threads
> >> may still be using that searcher.
> My solution is definitely not good if search requests must be served on that core of that server...
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-1070) IndexSearcher not closed after
indexing with DataImportHandler
Posted by "Yonik Seeley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-1070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12683209#action_12683209 ]
Yonik Seeley commented on SOLR-1070:
------------------------------------
I haven't been able to reproduce any problem with normal XML update commands (with multicore either)... seems likely related to DIH somehow.
Marc, what's the refCount on the core with too many searchers (check the admin stats page)? It should be "2"... one because the core is in service, and one for the request that is generating the stats page.
> IndexSearcher not closed after indexing with DataImportHandler
> --------------------------------------------------------------
>
> Key: SOLR-1070
> URL: https://issues.apache.org/jira/browse/SOLR-1070
> Project: Solr
> Issue Type: Bug
> Components: clients - java
> Affects Versions: 1.4
> Environment: Debian 2.6.26-1-xen-amd64
> Tomcat 5.5
> Reporter: Marc Sturlese
> Fix For: 1.4
>
>
> What I have noticed is afecting to the nightly build using multicore. I am using tomcat 5.5 server and Debian 2.6.26-1-xen-amd64.
> After indexing and optimizing with DataImporthandler the old IndexSearcher is not being proper closed.
> To reproduce the bug I have done a full-import from a mysql database optimizing at the end. Once done I change the data from the database and index optimizing again.
> When now I do a lsof | grep solr I can see that tomcat is holding the deleted files from old index.
> I can see the error in the stats screen aswell... if I do 4 full-imports, stats will show me 3 opened IndexSearchers.
> As I index on the master and never use it to serve search requests I have sorted it modifying the function getSearcher in SolrCore.java
> There's a part in the source where is coded:
> // we are all done with the old searcher we used
> // for warming...
> if (currSearcherHolderF!=null){currSearcherHolderF.decref();}
> I force there to close the indexSearcher and remvoe it from the RefCounted:
> // we are all done with the old searcher we used
> // for warming...
> if (currSearcherHolderF!=null){
> currSearcherHolderF.get().close();
> currSearcherHolderF.decref();
> _searchers.remove();
> }
> As Yonik told me:
> >> Forcing it to close at the point you did is unsafe since other threads
> >> may still be using that searcher.
> My solution is definitely not good if search requests must be served on that core of that server...
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-1070) IndexSearcher not closed after
indexing with DataImportHandler
Posted by "Marc Sturlese (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-1070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12683383#action_12683383 ]
Marc Sturlese commented on SOLR-1070:
-------------------------------------
Hi Noble, I think you show me the light.
Sorry for the loss of time, I think it was my fault.
In the transformer, once the index is done I save the Index Version in a database doing:
Context.getSolrCore().getSearcher().get().getReader().getVersion();
I thought it was just returning me the version and not opening a new searcher. After doing that I wasn't closing or decrementing anything.
So, can you give me an advice how to get the current searcher version in a correct way? I am not sure what should I decrement or close to avoid that problem...
Thanks and sorry again.
> IndexSearcher not closed after indexing with DataImportHandler
> --------------------------------------------------------------
>
> Key: SOLR-1070
> URL: https://issues.apache.org/jira/browse/SOLR-1070
> Project: Solr
> Issue Type: Bug
> Components: clients - java
> Affects Versions: 1.4
> Environment: Debian 2.6.26-1-xen-amd64
> Tomcat 5.5
> Reporter: Marc Sturlese
> Fix For: 1.4
>
>
> What I have noticed is afecting to the nightly build using multicore. I am using tomcat 5.5 server and Debian 2.6.26-1-xen-amd64.
> After indexing and optimizing with DataImporthandler the old IndexSearcher is not being proper closed.
> To reproduce the bug I have done a full-import from a mysql database optimizing at the end. Once done I change the data from the database and index optimizing again.
> When now I do a lsof | grep solr I can see that tomcat is holding the deleted files from old index.
> I can see the error in the stats screen aswell... if I do 4 full-imports, stats will show me 3 opened IndexSearchers.
> As I index on the master and never use it to serve search requests I have sorted it modifying the function getSearcher in SolrCore.java
> There's a part in the source where is coded:
> // we are all done with the old searcher we used
> // for warming...
> if (currSearcherHolderF!=null){currSearcherHolderF.decref();}
> I force there to close the indexSearcher and remvoe it from the RefCounted:
> // we are all done with the old searcher we used
> // for warming...
> if (currSearcherHolderF!=null){
> currSearcherHolderF.get().close();
> currSearcherHolderF.decref();
> _searchers.remove();
> }
> As Yonik told me:
> >> Forcing it to close at the point you did is unsafe since other threads
> >> may still be using that searcher.
> My solution is definitely not good if search requests must be served on that core of that server...
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-1070) IndexSearcher not closed after
indexing with DataImportHandler
Posted by "Shalin Shekhar Mangar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-1070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12683390#action_12683390 ]
Shalin Shekhar Mangar commented on SOLR-1070:
---------------------------------------------
Use it like this:
{code}
RefCounted<SolrIndexSearcher> searchHolder = null;
try {
searchHolder = Context.getSolrCore().getSearcher();
IndexReader reader = searchHolder.get().getReader();
// Do whatever you need to
} finally {
if (searchHolder != null) searchHolder.decref();
}
{code}
> IndexSearcher not closed after indexing with DataImportHandler
> --------------------------------------------------------------
>
> Key: SOLR-1070
> URL: https://issues.apache.org/jira/browse/SOLR-1070
> Project: Solr
> Issue Type: Bug
> Components: clients - java
> Affects Versions: 1.4
> Environment: Debian 2.6.26-1-xen-amd64
> Tomcat 5.5
> Reporter: Marc Sturlese
> Fix For: 1.4
>
>
> What I have noticed is afecting to the nightly build using multicore. I am using tomcat 5.5 server and Debian 2.6.26-1-xen-amd64.
> After indexing and optimizing with DataImporthandler the old IndexSearcher is not being proper closed.
> To reproduce the bug I have done a full-import from a mysql database optimizing at the end. Once done I change the data from the database and index optimizing again.
> When now I do a lsof | grep solr I can see that tomcat is holding the deleted files from old index.
> I can see the error in the stats screen aswell... if I do 4 full-imports, stats will show me 3 opened IndexSearchers.
> As I index on the master and never use it to serve search requests I have sorted it modifying the function getSearcher in SolrCore.java
> There's a part in the source where is coded:
> // we are all done with the old searcher we used
> // for warming...
> if (currSearcherHolderF!=null){currSearcherHolderF.decref();}
> I force there to close the indexSearcher and remvoe it from the RefCounted:
> // we are all done with the old searcher we used
> // for warming...
> if (currSearcherHolderF!=null){
> currSearcherHolderF.get().close();
> currSearcherHolderF.decref();
> _searchers.remove();
> }
> As Yonik told me:
> >> Forcing it to close at the point you did is unsafe since other threads
> >> may still be using that searcher.
> My solution is definitely not good if search requests must be served on that core of that server...
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-1070) IndexSearcher not closed after
indexing with DataImportHandler
Posted by "Marc Sturlese (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-1070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12683191#action_12683191 ]
Marc Sturlese commented on SOLR-1070:
-------------------------------------
Yes, I saw it uses CommitUpdateCommand, wich executes a commit from DirectUpdateHandler2.java. Haven't tried to index via xml with tomcat.
Seems that some code is not decrementing the reference decref() from SolrIndexSearcher once it's done
> IndexSearcher not closed after indexing with DataImportHandler
> --------------------------------------------------------------
>
> Key: SOLR-1070
> URL: https://issues.apache.org/jira/browse/SOLR-1070
> Project: Solr
> Issue Type: Bug
> Components: clients - java
> Affects Versions: 1.4
> Environment: Debian 2.6.26-1-xen-amd64
> Tomcat 5.5
> Reporter: Marc Sturlese
> Fix For: 1.4
>
>
> What I have noticed is afecting to the nightly build using multicore. I am using tomcat 5.5 server and Debian 2.6.26-1-xen-amd64.
> After indexing and optimizing with DataImporthandler the old IndexSearcher is not being proper closed.
> To reproduce the bug I have done a full-import from a mysql database optimizing at the end. Once done I change the data from the database and index optimizing again.
> When now I do a lsof | grep solr I can see that tomcat is holding the deleted files from old index.
> I can see the error in the stats screen aswell... if I do 4 full-imports, stats will show me 3 opened IndexSearchers.
> As I index on the master and never use it to serve search requests I have sorted it modifying the function getSearcher in SolrCore.java
> There's a part in the source where is coded:
> // we are all done with the old searcher we used
> // for warming...
> if (currSearcherHolderF!=null){currSearcherHolderF.decref();}
> I force there to close the indexSearcher and remvoe it from the RefCounted:
> // we are all done with the old searcher we used
> // for warming...
> if (currSearcherHolderF!=null){
> currSearcherHolderF.get().close();
> currSearcherHolderF.decref();
> _searchers.remove();
> }
> As Yonik told me:
> >> Forcing it to close at the point you did is unsafe since other threads
> >> may still be using that searcher.
> My solution is definitely not good if search requests must be served on that core of that server...
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (SOLR-1070) IndexSearcher not closed
after indexing with DataImportHandler
Posted by "Noble Paul (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-1070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12683314#action_12683314 ]
Noble Paul edited comment on SOLR-1070 at 3/18/09 9:38 PM:
-----------------------------------------------------------
this is the code which does a commit in DIH
{code:java}
//SolrWriter#commit()
CommitUpdateCommand commit = new CommitUpdateCommand(optimize);
updateProcessor.processCommit(commit);
{code}
is there anything else we need to do to fix this
DIH never uses an IndexSearcher. Do you have a Transformer or something which does a Context#SolrCore#getSearcher()
was (Author: noble.paul):
this is the code which does a commit in DIH
{code:java}
//SolrWriter#commit()
CommitUpdateCommand commit = new CommitUpdateCommand(optimize);
updateProcessor.processCommit(commit);
{code}
is there anything else we need to do to fix this
> IndexSearcher not closed after indexing with DataImportHandler
> --------------------------------------------------------------
>
> Key: SOLR-1070
> URL: https://issues.apache.org/jira/browse/SOLR-1070
> Project: Solr
> Issue Type: Bug
> Components: clients - java
> Affects Versions: 1.4
> Environment: Debian 2.6.26-1-xen-amd64
> Tomcat 5.5
> Reporter: Marc Sturlese
> Fix For: 1.4
>
>
> What I have noticed is afecting to the nightly build using multicore. I am using tomcat 5.5 server and Debian 2.6.26-1-xen-amd64.
> After indexing and optimizing with DataImporthandler the old IndexSearcher is not being proper closed.
> To reproduce the bug I have done a full-import from a mysql database optimizing at the end. Once done I change the data from the database and index optimizing again.
> When now I do a lsof | grep solr I can see that tomcat is holding the deleted files from old index.
> I can see the error in the stats screen aswell... if I do 4 full-imports, stats will show me 3 opened IndexSearchers.
> As I index on the master and never use it to serve search requests I have sorted it modifying the function getSearcher in SolrCore.java
> There's a part in the source where is coded:
> // we are all done with the old searcher we used
> // for warming...
> if (currSearcherHolderF!=null){currSearcherHolderF.decref();}
> I force there to close the indexSearcher and remvoe it from the RefCounted:
> // we are all done with the old searcher we used
> // for warming...
> if (currSearcherHolderF!=null){
> currSearcherHolderF.get().close();
> currSearcherHolderF.decref();
> _searchers.remove();
> }
> As Yonik told me:
> >> Forcing it to close at the point you did is unsafe since other threads
> >> may still be using that searcher.
> My solution is definitely not good if search requests must be served on that core of that server...
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.