You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Hoss Man (JIRA)" <ji...@apache.org> on 2011/07/15 01:37:59 UTC
[jira] [Created] (SOLR-2654) not used consistently in
all places Directory objects are instantiated
<lockType/> not used consistently in all places Directory objects are instantiated
----------------------------------------------------------------------------------
Key: SOLR-2654
URL: https://issues.apache.org/jira/browse/SOLR-2654
Project: Solr
Issue Type: Bug
Reporter: Hoss Man
Priority: Critical
Fix For: 3.4
nipunb noted on the mailing list then when configuring solr to use an alternate <lockType/> (ie: simple) the stats for the SolrIndexSearcher list NativeFSLockFactory being used by the Directory.
The problem seems to be that SolrIndexConfig is not consulted when constructing Directory objects used for IndexReader (it's only used by SolrIndexWriter)
I don't _think_ this is a problem in most cases since the IndexReaders should all be readOnly in the core solr code) but plugins could attempt to use them in other ways. In general it seems like a really bad bug waiting to happen.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (SOLR-2654) not used consistently in
all places Directory objects are instantiated
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mark Miller updated SOLR-2654:
------------------------------
Attachment: SOLR-2654.patch
So with this patch, tests pass for me - but it scares me because I had to change two places that used the std index path to use the newIndex path. That's odd stuff - need to figure out how this was working as is on trunk...
> <lockType/> not used consistently in all places Directory objects are instantiated
> ----------------------------------------------------------------------------------
>
> Key: SOLR-2654
> URL: https://issues.apache.org/jira/browse/SOLR-2654
> Project: Solr
> Issue Type: Bug
> Reporter: Hoss Man
> Assignee: Mark Miller
> Priority: Critical
> Fix For: 3.4
>
> Attachments: SOLR-2654.patch, SOLR-2654.patch
>
>
> nipunb noted on the mailing list then when configuring solr to use an alternate <lockType/> (ie: simple) the stats for the SolrIndexSearcher list NativeFSLockFactory being used by the Directory.
> The problem seems to be that SolrIndexConfig is not consulted when constructing Directory objects used for IndexReader (it's only used by SolrIndexWriter)
> I don't _think_ this is a problem in most cases since the IndexReaders should all be readOnly in the core solr code) but plugins could attempt to use them in other ways. In general it seems like a really bad bug waiting to happen.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (SOLR-2654) not used consistently in
all places Directory objects are instantiated
Posted by "Mark Miller (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mark Miller updated SOLR-2654:
------------------------------
Fix Version/s: (was: 3.6)
> <lockType/> not used consistently in all places Directory objects are instantiated
> ----------------------------------------------------------------------------------
>
> Key: SOLR-2654
> URL: https://issues.apache.org/jira/browse/SOLR-2654
> Project: Solr
> Issue Type: Bug
> Reporter: Hoss Man
> Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch, SOLR-2698.patch
>
>
> nipunb noted on the mailing list then when configuring solr to use an alternate <lockType/> (ie: simple) the stats for the SolrIndexSearcher list NativeFSLockFactory being used by the Directory.
> The problem seems to be that SolrIndexConfig is not consulted when constructing Directory objects used for IndexReader (it's only used by SolrIndexWriter)
> I don't _think_ this is a problem in most cases since the IndexReaders should all be readOnly in the core solr code) but plugins could attempt to use them in other ways. In general it seems like a really bad bug waiting to happen.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2654) not used consistently in
all places Directory objects are instantiated
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081021#comment-13081021 ]
Robert Muir commented on SOLR-2654:
-----------------------------------
Not sure its really a step backwards if tests pass without the test close directory hack!
couldn't we go with this as an iterative improvement, currently no directories are ever getting closed right?
> <lockType/> not used consistently in all places Directory objects are instantiated
> ----------------------------------------------------------------------------------
>
> Key: SOLR-2654
> URL: https://issues.apache.org/jira/browse/SOLR-2654
> Project: Solr
> Issue Type: Bug
> Reporter: Hoss Man
> Assignee: Mark Miller
> Fix For: 3.4, 4.0
>
> Attachments: SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch
>
>
> nipunb noted on the mailing list then when configuring solr to use an alternate <lockType/> (ie: simple) the stats for the SolrIndexSearcher list NativeFSLockFactory being used by the Directory.
> The problem seems to be that SolrIndexConfig is not consulted when constructing Directory objects used for IndexReader (it's only used by SolrIndexWriter)
> I don't _think_ this is a problem in most cases since the IndexReaders should all be readOnly in the core solr code) but plugins could attempt to use them in other ways. In general it seems like a really bad bug waiting to happen.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (SOLR-2654) not used consistently in
all places Directory objects are instantiated
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mark Miller updated SOLR-2654:
------------------------------
Attachment: SOLR-2654.patch
Okay, so first a step backwards - back to keeping the dirs around till the core is really closed, and no ref cnt dirs. Tests now pass without the test close directory hack.
I have not given up on something better, but I probably will take a break on this madness.
Eclipse has been doing weird stuff creating my patches on my Ubuntu machine - hopefully this one is legit, but I'll check it on my mac eventually to be sure.
> <lockType/> not used consistently in all places Directory objects are instantiated
> ----------------------------------------------------------------------------------
>
> Key: SOLR-2654
> URL: https://issues.apache.org/jira/browse/SOLR-2654
> Project: Solr
> Issue Type: Bug
> Reporter: Hoss Man
> Assignee: Mark Miller
> Fix For: 3.4, 4.0
>
> Attachments: SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch
>
>
> nipunb noted on the mailing list then when configuring solr to use an alternate <lockType/> (ie: simple) the stats for the SolrIndexSearcher list NativeFSLockFactory being used by the Directory.
> The problem seems to be that SolrIndexConfig is not consulted when constructing Directory objects used for IndexReader (it's only used by SolrIndexWriter)
> I don't _think_ this is a problem in most cases since the IndexReaders should all be readOnly in the core solr code) but plugins could attempt to use them in other ways. In general it seems like a really bad bug waiting to happen.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2654) not used consistently in
all places Directory objects are instantiated
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081024#comment-13081024 ]
Mark Miller commented on SOLR-2654:
-----------------------------------
bq. couldn't we go with this as an iterative improvement
Yeah, that's been on my mind - just didn't want to give up completely yet. If I commit this now, it will take the pressure off for figuring out something better. Give me a bit more time to defeat myself.
bq. currently no directories are ever getting closed right?
Right - but the tradeoff with this is, I think, is that before Directories would be garbage collected when they where no longer used - and now they will stick around till the SolrCore is really closed. Not a big deal for FSDirectory impls in general, but more troublesome for the ram based variety.
> <lockType/> not used consistently in all places Directory objects are instantiated
> ----------------------------------------------------------------------------------
>
> Key: SOLR-2654
> URL: https://issues.apache.org/jira/browse/SOLR-2654
> Project: Solr
> Issue Type: Bug
> Reporter: Hoss Man
> Assignee: Mark Miller
> Fix For: 3.4, 4.0
>
> Attachments: SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch
>
>
> nipunb noted on the mailing list then when configuring solr to use an alternate <lockType/> (ie: simple) the stats for the SolrIndexSearcher list NativeFSLockFactory being used by the Directory.
> The problem seems to be that SolrIndexConfig is not consulted when constructing Directory objects used for IndexReader (it's only used by SolrIndexWriter)
> I don't _think_ this is a problem in most cases since the IndexReaders should all be readOnly in the core solr code) but plugins could attempt to use them in other ways. In general it seems like a really bad bug waiting to happen.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (SOLR-2654) not used consistently in
all places Directory objects are instantiated
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mark Miller updated SOLR-2654:
------------------------------
Priority: Major (was: Critical)
> <lockType/> not used consistently in all places Directory objects are instantiated
> ----------------------------------------------------------------------------------
>
> Key: SOLR-2654
> URL: https://issues.apache.org/jira/browse/SOLR-2654
> Project: Solr
> Issue Type: Bug
> Reporter: Hoss Man
> Assignee: Mark Miller
> Fix For: 3.4, 4.0
>
> Attachments: SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch
>
>
> nipunb noted on the mailing list then when configuring solr to use an alternate <lockType/> (ie: simple) the stats for the SolrIndexSearcher list NativeFSLockFactory being used by the Directory.
> The problem seems to be that SolrIndexConfig is not consulted when constructing Directory objects used for IndexReader (it's only used by SolrIndexWriter)
> I don't _think_ this is a problem in most cases since the IndexReaders should all be readOnly in the core solr code) but plugins could attempt to use them in other ways. In general it seems like a really bad bug waiting to happen.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Comment Edited] (SOLR-2654) not used
consistently in all places Directory objects are instantiated
Posted by "Ronny Karallus (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13398874#comment-13398874 ]
Ronny Karallus edited comment on SOLR-2654 at 6/21/12 9:27 PM:
---------------------------------------------------------------
We just occured this problem using the spellchecking component. We got Solr 3.5 running.
Our configuration is:
...
<indexDefaults>
...
<lockType>simple</lockType>
</indexDefaults>
<searchComponent name="spellcheck" class="solr.SpellCheckComponent">
<str name="queryAnalyzerFieldType">textSpell</str>
<lst name="spellchecker">
<str name="name">default</str>
<str name="field">_spell</str>
<str name="spellcheckIndexDir">./spellchecker1</str>
<str name="buildOnCommit">true</str>
</lst>
...
</searchComponent>
That results in:
Caused by: org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: NativeFSLock@.../data/spellchecker1/write.lock
at org.apache.lucene.store.Lock.obtain(Lock.java:84)
at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1108)
at org.apache.lucene.search.spell.SpellChecker.setSpellIndex(SpellChecker.java:169)
at org.apache.lucene.search.spell.SpellChecker.<init>(SpellChecker.java:149)
at org.apache.solr.spelling.AbstractLuceneSpellChecker.init(AbstractLuceneSpellChecker.java:127)
... 22 more
Our FS doesnt support native locking. That's why we configured locktype to simple. However, the Spellchecker code doesn't use this setting ... That can be read pretty easy from the code:
Solr AbstractLuceneSpellChecker:
protected void initIndex() throws IOException {
if (indexDir != null) {
index = FSDirectory.open(new File(indexDir));
} else {
index = new RAMDirectory();
}
}
=> doesnt give a locking factory so the default is used
Lucene FSDirectory:
protected FSDirectory(File path, LockFactory lockFactory) throws IOException {
// new ctors use always NativeFSLockFactory as default:
if (lockFactory == null) {
lockFactory = new NativeFSLockFactory();
}
directory = getCanonicalPath(path);
if (directory.exists() && !directory.isDirectory())
throw new NoSuchDirectoryException("file '" + directory + "' exists but is not a directory");
setLockFactory(lockFactory);
}
=> the default is FNativeFSLock
was (Author: rkarallus):
We just occured this problem using the spellchecking component. We got Solr 3.5 running.
Our configuration is:
...
<indexDefaults>
...
<lockType>simple</lockType>
</indexDefaults>
<searchComponent name="spellcheck" class="solr.SpellCheckComponent">
<str name="queryAnalyzerFieldType">textSpell</str>
<lst name="spellchecker">
<str name="name">default</str>
<str name="field">_spell</str>
<str name="spellcheckIndexDir">./spellchecker1</str>
<str name="buildOnCommit">true</str>
</lst>
...
</searchComponent>
That results in:
Caused by: org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: NativeFSLock@.../data/spellchecker1/write.lock
at org.apache.lucene.store.Lock.obtain(Lock.java:84)
at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1108)
at org.apache.lucene.search.spell.SpellChecker.setSpellIndex(SpellChecker.java:169)
at org.apache.lucene.search.spell.SpellChecker.<init>(SpellChecker.java:149)
at org.apache.solr.spelling.AbstractLuceneSpellChecker.init(AbstractLuceneSpellChecker.java:127)
... 22 more
Our FS doesnt support native locking. That's why we configured locktype to simple. However, the Spellchecker code doesn't use this setting ...
> <lockType/> not used consistently in all places Directory objects are instantiated
> ----------------------------------------------------------------------------------
>
> Key: SOLR-2654
> URL: https://issues.apache.org/jira/browse/SOLR-2654
> Project: Solr
> Issue Type: Bug
> Reporter: Hoss Man
> Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch, SOLR-2698.patch
>
>
> nipunb noted on the mailing list then when configuring solr to use an alternate <lockType/> (ie: simple) the stats for the SolrIndexSearcher list NativeFSLockFactory being used by the Directory.
> The problem seems to be that SolrIndexConfig is not consulted when constructing Directory objects used for IndexReader (it's only used by SolrIndexWriter)
> I don't _think_ this is a problem in most cases since the IndexReaders should all be readOnly in the core solr code) but plugins could attempt to use them in other ways. In general it seems like a really bad bug waiting to happen.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2654) not used consistently in
all places Directory objects are instantiated
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13087106#comment-13087106 ]
Robert Muir commented on SOLR-2654:
-----------------------------------
+1
> <lockType/> not used consistently in all places Directory objects are instantiated
> ----------------------------------------------------------------------------------
>
> Key: SOLR-2654
> URL: https://issues.apache.org/jira/browse/SOLR-2654
> Project: Solr
> Issue Type: Bug
> Reporter: Hoss Man
> Assignee: Mark Miller
> Fix For: 3.4, 4.0
>
> Attachments: SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch, SOLR-2698.patch
>
>
> nipunb noted on the mailing list then when configuring solr to use an alternate <lockType/> (ie: simple) the stats for the SolrIndexSearcher list NativeFSLockFactory being used by the Directory.
> The problem seems to be that SolrIndexConfig is not consulted when constructing Directory objects used for IndexReader (it's only used by SolrIndexWriter)
> I don't _think_ this is a problem in most cases since the IndexReaders should all be readOnly in the core solr code) but plugins could attempt to use them in other ways. In general it seems like a really bad bug waiting to happen.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2654) not used consistently in
all places Directory objects are instantiated
Posted by "Ronny Karallus (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13398874#comment-13398874 ]
Ronny Karallus commented on SOLR-2654:
--------------------------------------
We just occured this problem using the spellchecking component. We got Solr 3.5 running.
Our configuration is:
...
<indexDefaults>
...
<lockType>simple</lockType>
</indexDefaults>
<searchComponent name="spellcheck" class="solr.SpellCheckComponent">
<str name="queryAnalyzerFieldType">textSpell</str>
<lst name="spellchecker">
<str name="name">default</str>
<str name="field">_spell</str>
<str name="spellcheckIndexDir">./spellchecker1</str>
<str name="buildOnCommit">true</str>
</lst>
...
</searchComponent>
That results in:
Caused by: org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: NativeFSLock@.../data/spellchecker1/write.lock
at org.apache.lucene.store.Lock.obtain(Lock.java:84)
at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1108)
at org.apache.lucene.search.spell.SpellChecker.setSpellIndex(SpellChecker.java:169)
at org.apache.lucene.search.spell.SpellChecker.<init>(SpellChecker.java:149)
at org.apache.solr.spelling.AbstractLuceneSpellChecker.init(AbstractLuceneSpellChecker.java:127)
... 22 more
Our FS doesnt support native locking. That's why we configured locktype to simple. However, the Spellchecker code doesn't use this setting ...
> <lockType/> not used consistently in all places Directory objects are instantiated
> ----------------------------------------------------------------------------------
>
> Key: SOLR-2654
> URL: https://issues.apache.org/jira/browse/SOLR-2654
> Project: Solr
> Issue Type: Bug
> Reporter: Hoss Man
> Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch, SOLR-2698.patch
>
>
> nipunb noted on the mailing list then when configuring solr to use an alternate <lockType/> (ie: simple) the stats for the SolrIndexSearcher list NativeFSLockFactory being used by the Directory.
> The problem seems to be that SolrIndexConfig is not consulted when constructing Directory objects used for IndexReader (it's only used by SolrIndexWriter)
> I don't _think_ this is a problem in most cases since the IndexReaders should all be readOnly in the core solr code) but plugins could attempt to use them in other ways. In general it seems like a really bad bug waiting to happen.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2654) not used consistently in
all places Directory objects are instantiated
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13080632#comment-13080632 ]
Mark Miller commented on SOLR-2654:
-----------------------------------
Nope, it was my bug. SolrIndexSearcher only supplies the right index dir if the directory is an instance of FSDirectory - since we now used a Directory wrapper, the index dir was not properly coming from the SolrIndexSearcher. This indexdir newIndexDir stuff is dicey ;)
Got it fixed locally, but I'll do a few more cleanup chores before supply another patch.
> <lockType/> not used consistently in all places Directory objects are instantiated
> ----------------------------------------------------------------------------------
>
> Key: SOLR-2654
> URL: https://issues.apache.org/jira/browse/SOLR-2654
> Project: Solr
> Issue Type: Bug
> Reporter: Hoss Man
> Assignee: Mark Miller
> Priority: Critical
> Fix For: 3.4, 4.0
>
> Attachments: SOLR-2654.patch, SOLR-2654.patch
>
>
> nipunb noted on the mailing list then when configuring solr to use an alternate <lockType/> (ie: simple) the stats for the SolrIndexSearcher list NativeFSLockFactory being used by the Directory.
> The problem seems to be that SolrIndexConfig is not consulted when constructing Directory objects used for IndexReader (it's only used by SolrIndexWriter)
> I don't _think_ this is a problem in most cases since the IndexReaders should all be readOnly in the core solr code) but plugins could attempt to use them in other ways. In general it seems like a really bad bug waiting to happen.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (SOLR-2654) not used consistently in
all places Directory objects are instantiated
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mark Miller updated SOLR-2654:
------------------------------
Attachment: SOLR-2654.patch
new patch:
*removes the refcnt dir wrappers
*cleans up a bunch of other nocommits / cleanup work
*adds a new NRTCachingDirectoryFactory - completely untested, but half the reason I'm working on improving this area ;) Uses hack ram / merge settings of 64. Needs to be set to reasonable defaults, plus be easily configurable eventually.
Tests pass for me.
> <lockType/> not used consistently in all places Directory objects are instantiated
> ----------------------------------------------------------------------------------
>
> Key: SOLR-2654
> URL: https://issues.apache.org/jira/browse/SOLR-2654
> Project: Solr
> Issue Type: Bug
> Reporter: Hoss Man
> Assignee: Mark Miller
> Priority: Critical
> Fix For: 3.4, 4.0
>
> Attachments: SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch
>
>
> nipunb noted on the mailing list then when configuring solr to use an alternate <lockType/> (ie: simple) the stats for the SolrIndexSearcher list NativeFSLockFactory being used by the Directory.
> The problem seems to be that SolrIndexConfig is not consulted when constructing Directory objects used for IndexReader (it's only used by SolrIndexWriter)
> I don't _think_ this is a problem in most cases since the IndexReaders should all be readOnly in the core solr code) but plugins could attempt to use them in other ways. In general it seems like a really bad bug waiting to happen.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (SOLR-2654) not used consistently in
all places Directory objects are instantiated
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mark Miller updated SOLR-2654:
------------------------------
Attachment: SOLR-2654.patch
I've cleaned up this patch quite a bit.
> <lockType/> not used consistently in all places Directory objects are instantiated
> ----------------------------------------------------------------------------------
>
> Key: SOLR-2654
> URL: https://issues.apache.org/jira/browse/SOLR-2654
> Project: Solr
> Issue Type: Bug
> Reporter: Hoss Man
> Assignee: Mark Miller
> Fix For: 3.4, 4.0
>
> Attachments: SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch, SOLR-2698.patch
>
>
> nipunb noted on the mailing list then when configuring solr to use an alternate <lockType/> (ie: simple) the stats for the SolrIndexSearcher list NativeFSLockFactory being used by the Directory.
> The problem seems to be that SolrIndexConfig is not consulted when constructing Directory objects used for IndexReader (it's only used by SolrIndexWriter)
> I don't _think_ this is a problem in most cases since the IndexReaders should all be readOnly in the core solr code) but plugins could attempt to use them in other ways. In general it seems like a really bad bug waiting to happen.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2654) not used consistently in
all places Directory objects are instantiated
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13080532#comment-13080532 ]
Mark Miller commented on SOLR-2654:
-----------------------------------
My initial suspicion is actually that these where bugs in trunk that where being hidden by the old behavior.
> <lockType/> not used consistently in all places Directory objects are instantiated
> ----------------------------------------------------------------------------------
>
> Key: SOLR-2654
> URL: https://issues.apache.org/jira/browse/SOLR-2654
> Project: Solr
> Issue Type: Bug
> Reporter: Hoss Man
> Assignee: Mark Miller
> Priority: Critical
> Fix For: 3.4
>
> Attachments: SOLR-2654.patch, SOLR-2654.patch
>
>
> nipunb noted on the mailing list then when configuring solr to use an alternate <lockType/> (ie: simple) the stats for the SolrIndexSearcher list NativeFSLockFactory being used by the Directory.
> The problem seems to be that SolrIndexConfig is not consulted when constructing Directory objects used for IndexReader (it's only used by SolrIndexWriter)
> I don't _think_ this is a problem in most cases since the IndexReaders should all be readOnly in the core solr code) but plugins could attempt to use them in other ways. In general it seems like a really bad bug waiting to happen.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2654) not used consistently in
all places Directory objects are instantiated
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13079477#comment-13079477 ]
Mark Miller commented on SOLR-2654:
-----------------------------------
So the lock issue is not really a problem - but not sharing the dir is depending on the impl as I mentioned.
I first tried tackling this outside of the directory factory - this quickly gets very difficult, most due to the newIndexPath that replication uses to avoid copying a full index.
I'm now exploring simply have the base directory factory cache all directories with the path as a key - how the ram directory factory works now.
> <lockType/> not used consistently in all places Directory objects are instantiated
> ----------------------------------------------------------------------------------
>
> Key: SOLR-2654
> URL: https://issues.apache.org/jira/browse/SOLR-2654
> Project: Solr
> Issue Type: Bug
> Reporter: Hoss Man
> Assignee: Mark Miller
> Priority: Critical
> Fix For: 3.4
>
>
> nipunb noted on the mailing list then when configuring solr to use an alternate <lockType/> (ie: simple) the stats for the SolrIndexSearcher list NativeFSLockFactory being used by the Directory.
> The problem seems to be that SolrIndexConfig is not consulted when constructing Directory objects used for IndexReader (it's only used by SolrIndexWriter)
> I don't _think_ this is a problem in most cases since the IndexReaders should all be readOnly in the core solr code) but plugins could attempt to use them in other ways. In general it seems like a really bad bug waiting to happen.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2654) not used consistently in
all places Directory objects are instantiated
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13080664#comment-13080664 ]
Robert Muir commented on SOLR-2654:
-----------------------------------
so we can remove the closeDirectories() hack in SolrTestCaseJ4?
> <lockType/> not used consistently in all places Directory objects are instantiated
> ----------------------------------------------------------------------------------
>
> Key: SOLR-2654
> URL: https://issues.apache.org/jira/browse/SOLR-2654
> Project: Solr
> Issue Type: Bug
> Reporter: Hoss Man
> Assignee: Mark Miller
> Fix For: 3.4, 4.0
>
> Attachments: SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch
>
>
> nipunb noted on the mailing list then when configuring solr to use an alternate <lockType/> (ie: simple) the stats for the SolrIndexSearcher list NativeFSLockFactory being used by the Directory.
> The problem seems to be that SolrIndexConfig is not consulted when constructing Directory objects used for IndexReader (it's only used by SolrIndexWriter)
> I don't _think_ this is a problem in most cases since the IndexReaders should all be readOnly in the core solr code) but plugins could attempt to use them in other ways. In general it seems like a really bad bug waiting to happen.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2654) not used consistently in
all places Directory objects are instantiated
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13080968#comment-13080968 ]
Mark Miller commented on SOLR-2654:
-----------------------------------
Yeesh - much more difficult to get this right than I expected. I may just go back to only closing when the SolrCore is finally closed - trying to get the searchers/writers to do the closing is just driving me nuts.
> <lockType/> not used consistently in all places Directory objects are instantiated
> ----------------------------------------------------------------------------------
>
> Key: SOLR-2654
> URL: https://issues.apache.org/jira/browse/SOLR-2654
> Project: Solr
> Issue Type: Bug
> Reporter: Hoss Man
> Assignee: Mark Miller
> Fix For: 3.4, 4.0
>
> Attachments: SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch
>
>
> nipunb noted on the mailing list then when configuring solr to use an alternate <lockType/> (ie: simple) the stats for the SolrIndexSearcher list NativeFSLockFactory being used by the Directory.
> The problem seems to be that SolrIndexConfig is not consulted when constructing Directory objects used for IndexReader (it's only used by SolrIndexWriter)
> I don't _think_ this is a problem in most cases since the IndexReaders should all be readOnly in the core solr code) but plugins could attempt to use them in other ways. In general it seems like a really bad bug waiting to happen.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2654) not used consistently in
all places Directory objects are instantiated
Posted by "Hoss Man (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13065614#comment-13065614 ]
Hoss Man commented on SOLR-2654:
--------------------------------
In general i wonder if there isn't a largely problem here: shouldn't the IndexReader and SolrIndexWriter usually be using the exact same _instance_ of a Directory object -- not just two instances generated by the same DirectoryFactory?)
(I'm not sure given the various work being done with reopen and what not)
> <lockType/> not used consistently in all places Directory objects are instantiated
> ----------------------------------------------------------------------------------
>
> Key: SOLR-2654
> URL: https://issues.apache.org/jira/browse/SOLR-2654
> Project: Solr
> Issue Type: Bug
> Reporter: Hoss Man
> Priority: Critical
> Fix For: 3.4
>
>
> nipunb noted on the mailing list then when configuring solr to use an alternate <lockType/> (ie: simple) the stats for the SolrIndexSearcher list NativeFSLockFactory being used by the Directory.
> The problem seems to be that SolrIndexConfig is not consulted when constructing Directory objects used for IndexReader (it's only used by SolrIndexWriter)
> I don't _think_ this is a problem in most cases since the IndexReaders should all be readOnly in the core solr code) but plugins could attempt to use them in other ways. In general it seems like a really bad bug waiting to happen.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Resolved] (SOLR-2654) not used consistently in
all places Directory objects are instantiated
Posted by "Mark Miller (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mark Miller resolved SOLR-2654.
-------------------------------
Resolution: Fixed
> <lockType/> not used consistently in all places Directory objects are instantiated
> ----------------------------------------------------------------------------------
>
> Key: SOLR-2654
> URL: https://issues.apache.org/jira/browse/SOLR-2654
> Project: Solr
> Issue Type: Bug
> Reporter: Hoss Man
> Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch, SOLR-2698.patch
>
>
> nipunb noted on the mailing list then when configuring solr to use an alternate <lockType/> (ie: simple) the stats for the SolrIndexSearcher list NativeFSLockFactory being used by the Directory.
> The problem seems to be that SolrIndexConfig is not consulted when constructing Directory objects used for IndexReader (it's only used by SolrIndexWriter)
> I don't _think_ this is a problem in most cases since the IndexReaders should all be readOnly in the core solr code) but plugins could attempt to use them in other ways. In general it seems like a really bad bug waiting to happen.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2654) not used consistently in
all places Directory objects are instantiated
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13080634#comment-13080634 ]
Mark Miller commented on SOLR-2654:
-----------------------------------
I'm actually going to kill this RefCntDirectory now - I had it because I was copying our RamDirectoryFactory hack - but the way things have changed, I don't think we need it at all anymore (not that we where closing Directories anyway - I think we actually can close them properly now though)
> <lockType/> not used consistently in all places Directory objects are instantiated
> ----------------------------------------------------------------------------------
>
> Key: SOLR-2654
> URL: https://issues.apache.org/jira/browse/SOLR-2654
> Project: Solr
> Issue Type: Bug
> Reporter: Hoss Man
> Assignee: Mark Miller
> Priority: Critical
> Fix For: 3.4, 4.0
>
> Attachments: SOLR-2654.patch, SOLR-2654.patch
>
>
> nipunb noted on the mailing list then when configuring solr to use an alternate <lockType/> (ie: simple) the stats for the SolrIndexSearcher list NativeFSLockFactory being used by the Directory.
> The problem seems to be that SolrIndexConfig is not consulted when constructing Directory objects used for IndexReader (it's only used by SolrIndexWriter)
> I don't _think_ this is a problem in most cases since the IndexReaders should all be readOnly in the core solr code) but plugins could attempt to use them in other ways. In general it seems like a really bad bug waiting to happen.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2654) not used consistently in
all places Directory objects are instantiated
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081466#comment-13081466 ]
Mark Miller commented on SOLR-2654:
-----------------------------------
Okay, looks like i am close to having this figured out - I changed tactics to do the ref counting in the dir factory, and then having the user release the directory. For whatever reason, this made things easier to debug. I found an issue I had that was making the previous attempt difficult, and also what looks like a nasty bug (more to come) that was compounding the issue. Still a fair amount to try and do to finish this off as I want, but making progress.
> <lockType/> not used consistently in all places Directory objects are instantiated
> ----------------------------------------------------------------------------------
>
> Key: SOLR-2654
> URL: https://issues.apache.org/jira/browse/SOLR-2654
> Project: Solr
> Issue Type: Bug
> Reporter: Hoss Man
> Assignee: Mark Miller
> Fix For: 3.4, 4.0
>
> Attachments: SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch
>
>
> nipunb noted on the mailing list then when configuring solr to use an alternate <lockType/> (ie: simple) the stats for the SolrIndexSearcher list NativeFSLockFactory being used by the Directory.
> The problem seems to be that SolrIndexConfig is not consulted when constructing Directory objects used for IndexReader (it's only used by SolrIndexWriter)
> I don't _think_ this is a problem in most cases since the IndexReaders should all be readOnly in the core solr code) but plugins could attempt to use them in other ways. In general it seems like a really bad bug waiting to happen.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2654) not used consistently in
all places Directory objects are instantiated
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13086709#comment-13086709 ]
Robert Muir commented on SOLR-2654:
-----------------------------------
{quote}
I take the ensureOpen check out of IndexWriter on getDirectory. Anyone object? I don't know why we care if the IndexWriter is still open if we want to grab which Directory it used. You can hack around it, but I don't see why we don't just remove the check. Thoughts?
{quote}
+1... its just a getter for the Dir, no reason to ensure its open. you might just want to do instanceof or something.
In any case, the Directory *itself* not indexwriter should be calling ensureOpen before doing things that require an open dir.
Maybe we could spin off a separate issue to make a test for this, i know its inconsistent everywhere.
> <lockType/> not used consistently in all places Directory objects are instantiated
> ----------------------------------------------------------------------------------
>
> Key: SOLR-2654
> URL: https://issues.apache.org/jira/browse/SOLR-2654
> Project: Solr
> Issue Type: Bug
> Reporter: Hoss Man
> Assignee: Mark Miller
> Fix For: 3.4, 4.0
>
> Attachments: SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch, SOLR-2698.patch
>
>
> nipunb noted on the mailing list then when configuring solr to use an alternate <lockType/> (ie: simple) the stats for the SolrIndexSearcher list NativeFSLockFactory being used by the Directory.
> The problem seems to be that SolrIndexConfig is not consulted when constructing Directory objects used for IndexReader (it's only used by SolrIndexWriter)
> I don't _think_ this is a problem in most cases since the IndexReaders should all be readOnly in the core solr code) but plugins could attempt to use them in other ways. In general it seems like a really bad bug waiting to happen.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (SOLR-2654) not used consistently in
all places Directory objects are instantiated
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mark Miller updated SOLR-2654:
------------------------------
Attachment: SOLR-2698.patch
Latest rough patch
> <lockType/> not used consistently in all places Directory objects are instantiated
> ----------------------------------------------------------------------------------
>
> Key: SOLR-2654
> URL: https://issues.apache.org/jira/browse/SOLR-2654
> Project: Solr
> Issue Type: Bug
> Reporter: Hoss Man
> Assignee: Mark Miller
> Fix For: 3.4, 4.0
>
> Attachments: SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch, SOLR-2698.patch
>
>
> nipunb noted on the mailing list then when configuring solr to use an alternate <lockType/> (ie: simple) the stats for the SolrIndexSearcher list NativeFSLockFactory being used by the Directory.
> The problem seems to be that SolrIndexConfig is not consulted when constructing Directory objects used for IndexReader (it's only used by SolrIndexWriter)
> I don't _think_ this is a problem in most cases since the IndexReaders should all be readOnly in the core solr code) but plugins could attempt to use them in other ways. In general it seems like a really bad bug waiting to happen.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2654) not used consistently in
all places Directory objects are instantiated
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13080662#comment-13080662 ]
Mark Miller commented on SOLR-2654:
-----------------------------------
So I have one last nocommit - directories pile up in the cache over time, and are only released when the SolrCore is finally truly closed (usually end of program). Ideally, we like to close them earlier when we can, and pull them from the cache (think clearing a RamDirectory).
So I think we can fix this by having SolrIndexSearcher and SolrIndexWriter close their directories in their close methods - of course that likely brings back my friend the ref cnt directory ;)
> <lockType/> not used consistently in all places Directory objects are instantiated
> ----------------------------------------------------------------------------------
>
> Key: SOLR-2654
> URL: https://issues.apache.org/jira/browse/SOLR-2654
> Project: Solr
> Issue Type: Bug
> Reporter: Hoss Man
> Assignee: Mark Miller
> Fix For: 3.4, 4.0
>
> Attachments: SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch
>
>
> nipunb noted on the mailing list then when configuring solr to use an alternate <lockType/> (ie: simple) the stats for the SolrIndexSearcher list NativeFSLockFactory being used by the Directory.
> The problem seems to be that SolrIndexConfig is not consulted when constructing Directory objects used for IndexReader (it's only used by SolrIndexWriter)
> I don't _think_ this is a problem in most cases since the IndexReaders should all be readOnly in the core solr code) but plugins could attempt to use them in other ways. In general it seems like a really bad bug waiting to happen.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2654) not used consistently in
all places Directory objects are instantiated
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13080525#comment-13080525 ]
Mark Miller commented on SOLR-2654:
-----------------------------------
Hmmm...the problem has something to do with this new index stuff that replication uses - this thing always gets in my way :)
> <lockType/> not used consistently in all places Directory objects are instantiated
> ----------------------------------------------------------------------------------
>
> Key: SOLR-2654
> URL: https://issues.apache.org/jira/browse/SOLR-2654
> Project: Solr
> Issue Type: Bug
> Reporter: Hoss Man
> Assignee: Mark Miller
> Priority: Critical
> Fix For: 3.4
>
> Attachments: SOLR-2654.patch
>
>
> nipunb noted on the mailing list then when configuring solr to use an alternate <lockType/> (ie: simple) the stats for the SolrIndexSearcher list NativeFSLockFactory being used by the Directory.
> The problem seems to be that SolrIndexConfig is not consulted when constructing Directory objects used for IndexReader (it's only used by SolrIndexWriter)
> I don't _think_ this is a problem in most cases since the IndexReaders should all be readOnly in the core solr code) but plugins could attempt to use them in other ways. In general it seems like a really bad bug waiting to happen.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (SOLR-2654) not used consistently in
all places Directory objects are instantiated
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mark Miller updated SOLR-2654:
------------------------------
Attachment: SOLR-2654.patch
Another, more cleaned up, patch.
> <lockType/> not used consistently in all places Directory objects are instantiated
> ----------------------------------------------------------------------------------
>
> Key: SOLR-2654
> URL: https://issues.apache.org/jira/browse/SOLR-2654
> Project: Solr
> Issue Type: Bug
> Reporter: Hoss Man
> Assignee: Mark Miller
> Fix For: 3.4, 4.0
>
> Attachments: SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch, SOLR-2698.patch
>
>
> nipunb noted on the mailing list then when configuring solr to use an alternate <lockType/> (ie: simple) the stats for the SolrIndexSearcher list NativeFSLockFactory being used by the Directory.
> The problem seems to be that SolrIndexConfig is not consulted when constructing Directory objects used for IndexReader (it's only used by SolrIndexWriter)
> I don't _think_ this is a problem in most cases since the IndexReaders should all be readOnly in the core solr code) but plugins could attempt to use them in other ways. In general it seems like a really bad bug waiting to happen.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2654) not used consistently in
all places Directory objects are instantiated
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13072527#comment-13072527 ]
Mark Miller commented on SOLR-2654:
-----------------------------------
I think practically it's not a large problem - since the write lock is used for IndexWriter only as you say. Other than that, it would depend on the lock factory impl. It also depends on the Directory impl. I think in general, current impls work fine this way.
We should address this I think though - we shouldn't count on this to work correctly.
It's tricky in a couple areas though. I'll address on trunk before considering 3.4.
> <lockType/> not used consistently in all places Directory objects are instantiated
> ----------------------------------------------------------------------------------
>
> Key: SOLR-2654
> URL: https://issues.apache.org/jira/browse/SOLR-2654
> Project: Solr
> Issue Type: Bug
> Reporter: Hoss Man
> Assignee: Mark Miller
> Priority: Critical
> Fix For: 3.4
>
>
> nipunb noted on the mailing list then when configuring solr to use an alternate <lockType/> (ie: simple) the stats for the SolrIndexSearcher list NativeFSLockFactory being used by the Directory.
> The problem seems to be that SolrIndexConfig is not consulted when constructing Directory objects used for IndexReader (it's only used by SolrIndexWriter)
> I don't _think_ this is a problem in most cases since the IndexReaders should all be readOnly in the core solr code) but plugins could attempt to use them in other ways. In general it seems like a really bad bug waiting to happen.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2654) not used consistently in
all places Directory objects are instantiated
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13080687#comment-13080687 ]
Mark Miller commented on SOLR-2654:
-----------------------------------
So I removed the close directories hack for tests and realized that I need the solution I proposed above regardless - when we force open a new directory, we need to close the old one - because we replace it in the map (well, close it, or keep it around in another list/set). We cannot close it right then, because something still might be using it. The ref cnt dir is def coming back.
> <lockType/> not used consistently in all places Directory objects are instantiated
> ----------------------------------------------------------------------------------
>
> Key: SOLR-2654
> URL: https://issues.apache.org/jira/browse/SOLR-2654
> Project: Solr
> Issue Type: Bug
> Reporter: Hoss Man
> Assignee: Mark Miller
> Fix For: 3.4, 4.0
>
> Attachments: SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch
>
>
> nipunb noted on the mailing list then when configuring solr to use an alternate <lockType/> (ie: simple) the stats for the SolrIndexSearcher list NativeFSLockFactory being used by the Directory.
> The problem seems to be that SolrIndexConfig is not consulted when constructing Directory objects used for IndexReader (it's only used by SolrIndexWriter)
> I don't _think_ this is a problem in most cases since the IndexReaders should all be readOnly in the core solr code) but plugins could attempt to use them in other ways. In general it seems like a really bad bug waiting to happen.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (SOLR-2654) not used consistently in
all places Directory objects are instantiated
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mark Miller updated SOLR-2654:
------------------------------
Attachment: SOLR-2654.patch
new patch - the problem with pure ref counting is that sometimes you might hit a point where you don't need the directory for a moment - and then you do - if its a ram based dir that means you lose your dir data. So this is something of a hybrid - it still ref counts, but only when a new index writer is forced open does it allow a directory to actually be closed. So when the indexwriter jumps to a new index, that should allow the old one to be removed once all the refs are released. The current working directory will always be kept around. All created directories are are tracked so that any that are not closed can be closed when the SolrCore is actually closed (not reloaded).
All tests currently passing.
> <lockType/> not used consistently in all places Directory objects are instantiated
> ----------------------------------------------------------------------------------
>
> Key: SOLR-2654
> URL: https://issues.apache.org/jira/browse/SOLR-2654
> Project: Solr
> Issue Type: Bug
> Reporter: Hoss Man
> Assignee: Mark Miller
> Fix For: 3.4, 4.0
>
> Attachments: SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch, SOLR-2698.patch
>
>
> nipunb noted on the mailing list then when configuring solr to use an alternate <lockType/> (ie: simple) the stats for the SolrIndexSearcher list NativeFSLockFactory being used by the Directory.
> The problem seems to be that SolrIndexConfig is not consulted when constructing Directory objects used for IndexReader (it's only used by SolrIndexWriter)
> I don't _think_ this is a problem in most cases since the IndexReaders should all be readOnly in the core solr code) but plugins could attempt to use them in other ways. In general it seems like a really bad bug waiting to happen.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2654) not used consistently in
all places Directory objects are instantiated
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13086618#comment-13086618 ]
Mark Miller commented on SOLR-2654:
-----------------------------------
This makes one change in lucene that I could break out into it's own issue, but it's rather small - so I will collect any opinions here for now:
I take the ensureOpen check out of IndexWriter on getDirectory. Anyone object? I don't know why we care if the IndexWriter is still open if we want to grab which Directory it used. You can hack around it, but I don't see why we don't just remove the check. Thoughts?
> <lockType/> not used consistently in all places Directory objects are instantiated
> ----------------------------------------------------------------------------------
>
> Key: SOLR-2654
> URL: https://issues.apache.org/jira/browse/SOLR-2654
> Project: Solr
> Issue Type: Bug
> Reporter: Hoss Man
> Assignee: Mark Miller
> Fix For: 3.4, 4.0
>
> Attachments: SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch, SOLR-2698.patch
>
>
> nipunb noted on the mailing list then when configuring solr to use an alternate <lockType/> (ie: simple) the stats for the SolrIndexSearcher list NativeFSLockFactory being used by the Directory.
> The problem seems to be that SolrIndexConfig is not consulted when constructing Directory objects used for IndexReader (it's only used by SolrIndexWriter)
> I don't _think_ this is a problem in most cases since the IndexReaders should all be readOnly in the core solr code) but plugins could attempt to use them in other ways. In general it seems like a really bad bug waiting to happen.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2654) not used consistently in
all places Directory objects are instantiated
Posted by "Mark Miller (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13177259#comment-13177259 ]
Mark Miller commented on SOLR-2654:
-----------------------------------
I don't think there is enough gain to tackle something on the 3x branch here. If further info changes that, or someone else wants to tackle it, we should just open a new issue. I'm going to resolve this for 4 since we have a solution there.
> <lockType/> not used consistently in all places Directory objects are instantiated
> ----------------------------------------------------------------------------------
>
> Key: SOLR-2654
> URL: https://issues.apache.org/jira/browse/SOLR-2654
> Project: Solr
> Issue Type: Bug
> Reporter: Hoss Man
> Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch, SOLR-2698.patch
>
>
> nipunb noted on the mailing list then when configuring solr to use an alternate <lockType/> (ie: simple) the stats for the SolrIndexSearcher list NativeFSLockFactory being used by the Directory.
> The problem seems to be that SolrIndexConfig is not consulted when constructing Directory objects used for IndexReader (it's only used by SolrIndexWriter)
> I don't _think_ this is a problem in most cases since the IndexReaders should all be readOnly in the core solr code) but plugins could attempt to use them in other ways. In general it seems like a really bad bug waiting to happen.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2654) not used consistently in
all places Directory objects are instantiated
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13080669#comment-13080669 ]
Mark Miller commented on SOLR-2654:
-----------------------------------
Current patch, directories are all closed when a SolrCore is finally closed (not reloaded). This is new, and should alleviate the need for the test close dir hack.
The reason I want to sometimes close them early is that the index dir for a core can change - either on reload (unlikely) or by the replication handler. So if you had a Solr instance up for a long time, and used replication, you could end up, over time, with a lot of open Directories in the directory cache - currently, this is not much of a problem - replication only works with FSDirectories anyway. But I'd like to see that limitation fixed too - so that something like a RAMDirectory could work with replication - and in that case, this would be slightly more important*, as an unclosed RAMDirectory will take up much more RAM than an FSDirectory, which does nothing on close anyway (*only slightly more important because replication only uses a new index dir in a minority of cases).
> <lockType/> not used consistently in all places Directory objects are instantiated
> ----------------------------------------------------------------------------------
>
> Key: SOLR-2654
> URL: https://issues.apache.org/jira/browse/SOLR-2654
> Project: Solr
> Issue Type: Bug
> Reporter: Hoss Man
> Assignee: Mark Miller
> Fix For: 3.4, 4.0
>
> Attachments: SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch
>
>
> nipunb noted on the mailing list then when configuring solr to use an alternate <lockType/> (ie: simple) the stats for the SolrIndexSearcher list NativeFSLockFactory being used by the Directory.
> The problem seems to be that SolrIndexConfig is not consulted when constructing Directory objects used for IndexReader (it's only used by SolrIndexWriter)
> I don't _think_ this is a problem in most cases since the IndexReaders should all be readOnly in the core solr code) but plugins could attempt to use them in other ways. In general it seems like a really bad bug waiting to happen.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2654) not used consistently in
all places Directory objects are instantiated
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081065#comment-13081065 ]
Mark Miller commented on SOLR-2654:
-----------------------------------
P.S. to clarify, by step backwards, I meant vs the approach I was banging my head against. It's def a step forward from trunk mostly, and a step forward from the last patch (it now keeps all dirs seen in a list, so none are lost by the time we close them all).
> <lockType/> not used consistently in all places Directory objects are instantiated
> ----------------------------------------------------------------------------------
>
> Key: SOLR-2654
> URL: https://issues.apache.org/jira/browse/SOLR-2654
> Project: Solr
> Issue Type: Bug
> Reporter: Hoss Man
> Assignee: Mark Miller
> Fix For: 3.4, 4.0
>
> Attachments: SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch
>
>
> nipunb noted on the mailing list then when configuring solr to use an alternate <lockType/> (ie: simple) the stats for the SolrIndexSearcher list NativeFSLockFactory being used by the Directory.
> The problem seems to be that SolrIndexConfig is not consulted when constructing Directory objects used for IndexReader (it's only used by SolrIndexWriter)
> I don't _think_ this is a problem in most cases since the IndexReaders should all be readOnly in the core solr code) but plugins could attempt to use them in other ways. In general it seems like a really bad bug waiting to happen.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2654) not used consistently in
all places Directory objects are instantiated
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13087241#comment-13087241 ]
Mark Miller commented on SOLR-2654:
-----------------------------------
Committed to trunk.
This cannot be done the same way on 3.x. 3.x does not have a mechanism to pass state across core reloads, and that's a change I'd rather leave to 4.x.
Might simply be best to track all directories and close them on shutdown.
> <lockType/> not used consistently in all places Directory objects are instantiated
> ----------------------------------------------------------------------------------
>
> Key: SOLR-2654
> URL: https://issues.apache.org/jira/browse/SOLR-2654
> Project: Solr
> Issue Type: Bug
> Reporter: Hoss Man
> Assignee: Mark Miller
> Fix For: 3.4, 4.0
>
> Attachments: SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch, SOLR-2698.patch
>
>
> nipunb noted on the mailing list then when configuring solr to use an alternate <lockType/> (ie: simple) the stats for the SolrIndexSearcher list NativeFSLockFactory being used by the Directory.
> The problem seems to be that SolrIndexConfig is not consulted when constructing Directory objects used for IndexReader (it's only used by SolrIndexWriter)
> I don't _think_ this is a problem in most cases since the IndexReaders should all be readOnly in the core solr code) but plugins could attempt to use them in other ways. In general it seems like a really bad bug waiting to happen.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (SOLR-2654) not used consistently in
all places Directory objects are instantiated
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mark Miller updated SOLR-2654:
------------------------------
Attachment: SOLR-2654.patch
first rough patch -
TestReplicationHandler does not pass -
doTestReplicateAfterWrite2Slave causes any of the other replication tests run after it to fail. Commenting it out, all tests pass.
This allows us to use the same dir instance everywhere (kind of important - we hacked around this for ram dir specifically by using a static cache - mainly for tests).
It does kind of make it so you must use the new approved SolrCore reload method rather than just starting a new clore and closing the old one. But you really should do that anyhow, so...and it's better than using a static cache IMO.
> <lockType/> not used consistently in all places Directory objects are instantiated
> ----------------------------------------------------------------------------------
>
> Key: SOLR-2654
> URL: https://issues.apache.org/jira/browse/SOLR-2654
> Project: Solr
> Issue Type: Bug
> Reporter: Hoss Man
> Assignee: Mark Miller
> Priority: Critical
> Fix For: 3.4
>
> Attachments: SOLR-2654.patch
>
>
> nipunb noted on the mailing list then when configuring solr to use an alternate <lockType/> (ie: simple) the stats for the SolrIndexSearcher list NativeFSLockFactory being used by the Directory.
> The problem seems to be that SolrIndexConfig is not consulted when constructing Directory objects used for IndexReader (it's only used by SolrIndexWriter)
> I don't _think_ this is a problem in most cases since the IndexReaders should all be readOnly in the core solr code) but plugins could attempt to use them in other ways. In general it seems like a really bad bug waiting to happen.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Comment Edited] (SOLR-2654) not used
consistently in all places Directory objects are instantiated
Posted by "Ronny Karallus (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13398874#comment-13398874 ]
Ronny Karallus edited comment on SOLR-2654 at 6/21/12 9:28 PM:
---------------------------------------------------------------
We just occured this problem using the spellchecking component. We got Solr 3.5 running.
Our configuration is:
...
<indexDefaults>
...
<lockType>simple</lockType>
</indexDefaults>
<searchComponent name="spellcheck" class="solr.SpellCheckComponent">
<str name="queryAnalyzerFieldType">textSpell</str>
<lst name="spellchecker">
<str name="name">default</str>
<str name="field">_spell</str>
<str name="spellcheckIndexDir">./spellchecker1</str>
<str name="buildOnCommit">true</str>
</lst>
...
</searchComponent>
That results in:
Caused by: org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: NativeFSLock@.../data/spellchecker1/write.lock
at org.apache.lucene.store.Lock.obtain(Lock.java:84)
at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1108)
at org.apache.lucene.search.spell.SpellChecker.setSpellIndex(SpellChecker.java:169)
at org.apache.lucene.search.spell.SpellChecker.<init>(SpellChecker.java:149)
at org.apache.solr.spelling.AbstractLuceneSpellChecker.init(AbstractLuceneSpellChecker.java:127)
... 22 more
Our FS doesnt support native locking. That's why we configured locktype to simple. However, the Spellchecker code doesn't use this setting ... That can be read pretty easy from the code:
Solr AbstractLuceneSpellChecker:
protected void initIndex() throws IOException {
if (indexDir != null) {
index = FSDirectory.open(new File(indexDir));
} else {
index = new RAMDirectory();
}
}
=> doesnt give a locking factory so the default is used
Lucene FSDirectory:
protected FSDirectory(File path, LockFactory lockFactory) throws IOException {
// new ctors use always NativeFSLockFactory as default:
if (lockFactory == null) {
lockFactory = new NativeFSLockFactory();
}
directory = getCanonicalPath(path);
if (directory.exists() && !directory.isDirectory())
throw new NoSuchDirectoryException("file '" + directory + "' exists but is not a directory");
setLockFactory(lockFactory);
}
=> the default is NativeFSLock
I think that makes the problem a little less theoretical and more severe then the read scenario
was (Author: rkarallus):
We just occured this problem using the spellchecking component. We got Solr 3.5 running.
Our configuration is:
...
<indexDefaults>
...
<lockType>simple</lockType>
</indexDefaults>
<searchComponent name="spellcheck" class="solr.SpellCheckComponent">
<str name="queryAnalyzerFieldType">textSpell</str>
<lst name="spellchecker">
<str name="name">default</str>
<str name="field">_spell</str>
<str name="spellcheckIndexDir">./spellchecker1</str>
<str name="buildOnCommit">true</str>
</lst>
...
</searchComponent>
That results in:
Caused by: org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: NativeFSLock@.../data/spellchecker1/write.lock
at org.apache.lucene.store.Lock.obtain(Lock.java:84)
at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1108)
at org.apache.lucene.search.spell.SpellChecker.setSpellIndex(SpellChecker.java:169)
at org.apache.lucene.search.spell.SpellChecker.<init>(SpellChecker.java:149)
at org.apache.solr.spelling.AbstractLuceneSpellChecker.init(AbstractLuceneSpellChecker.java:127)
... 22 more
Our FS doesnt support native locking. That's why we configured locktype to simple. However, the Spellchecker code doesn't use this setting ... That can be read pretty easy from the code:
Solr AbstractLuceneSpellChecker:
protected void initIndex() throws IOException {
if (indexDir != null) {
index = FSDirectory.open(new File(indexDir));
} else {
index = new RAMDirectory();
}
}
=> doesnt give a locking factory so the default is used
Lucene FSDirectory:
protected FSDirectory(File path, LockFactory lockFactory) throws IOException {
// new ctors use always NativeFSLockFactory as default:
if (lockFactory == null) {
lockFactory = new NativeFSLockFactory();
}
directory = getCanonicalPath(path);
if (directory.exists() && !directory.isDirectory())
throw new NoSuchDirectoryException("file '" + directory + "' exists but is not a directory");
setLockFactory(lockFactory);
}
=> the default is FNativeFSLock
> <lockType/> not used consistently in all places Directory objects are instantiated
> ----------------------------------------------------------------------------------
>
> Key: SOLR-2654
> URL: https://issues.apache.org/jira/browse/SOLR-2654
> Project: Solr
> Issue Type: Bug
> Reporter: Hoss Man
> Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch, SOLR-2698.patch
>
>
> nipunb noted on the mailing list then when configuring solr to use an alternate <lockType/> (ie: simple) the stats for the SolrIndexSearcher list NativeFSLockFactory being used by the Directory.
> The problem seems to be that SolrIndexConfig is not consulted when constructing Directory objects used for IndexReader (it's only used by SolrIndexWriter)
> I don't _think_ this is a problem in most cases since the IndexReaders should all be readOnly in the core solr code) but plugins could attempt to use them in other ways. In general it seems like a really bad bug waiting to happen.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Assigned] (SOLR-2654) not used consistently in
all places Directory objects are instantiated
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mark Miller reassigned SOLR-2654:
---------------------------------
Assignee: Mark Miller
> <lockType/> not used consistently in all places Directory objects are instantiated
> ----------------------------------------------------------------------------------
>
> Key: SOLR-2654
> URL: https://issues.apache.org/jira/browse/SOLR-2654
> Project: Solr
> Issue Type: Bug
> Reporter: Hoss Man
> Assignee: Mark Miller
> Priority: Critical
> Fix For: 3.4
>
>
> nipunb noted on the mailing list then when configuring solr to use an alternate <lockType/> (ie: simple) the stats for the SolrIndexSearcher list NativeFSLockFactory being used by the Directory.
> The problem seems to be that SolrIndexConfig is not consulted when constructing Directory objects used for IndexReader (it's only used by SolrIndexWriter)
> I don't _think_ this is a problem in most cases since the IndexReaders should all be readOnly in the core solr code) but plugins could attempt to use them in other ways. In general it seems like a really bad bug waiting to happen.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2654) not used consistently in
all places Directory objects are instantiated
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13080668#comment-13080668 ]
Mark Miller commented on SOLR-2654:
-----------------------------------
Yeah, we def should be able to after this.
> <lockType/> not used consistently in all places Directory objects are instantiated
> ----------------------------------------------------------------------------------
>
> Key: SOLR-2654
> URL: https://issues.apache.org/jira/browse/SOLR-2654
> Project: Solr
> Issue Type: Bug
> Reporter: Hoss Man
> Assignee: Mark Miller
> Fix For: 3.4, 4.0
>
> Attachments: SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch
>
>
> nipunb noted on the mailing list then when configuring solr to use an alternate <lockType/> (ie: simple) the stats for the SolrIndexSearcher list NativeFSLockFactory being used by the Directory.
> The problem seems to be that SolrIndexConfig is not consulted when constructing Directory objects used for IndexReader (it's only used by SolrIndexWriter)
> I don't _think_ this is a problem in most cases since the IndexReaders should all be readOnly in the core solr code) but plugins could attempt to use them in other ways. In general it seems like a really bad bug waiting to happen.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2654) not used consistently in
all places Directory objects are instantiated
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13086999#comment-13086999 ]
Mark Miller commented on SOLR-2654:
-----------------------------------
Cool - I'm growing tempted to commit this then.
> <lockType/> not used consistently in all places Directory objects are instantiated
> ----------------------------------------------------------------------------------
>
> Key: SOLR-2654
> URL: https://issues.apache.org/jira/browse/SOLR-2654
> Project: Solr
> Issue Type: Bug
> Reporter: Hoss Man
> Assignee: Mark Miller
> Fix For: 3.4, 4.0
>
> Attachments: SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch, SOLR-2698.patch
>
>
> nipunb noted on the mailing list then when configuring solr to use an alternate <lockType/> (ie: simple) the stats for the SolrIndexSearcher list NativeFSLockFactory being used by the Directory.
> The problem seems to be that SolrIndexConfig is not consulted when constructing Directory objects used for IndexReader (it's only used by SolrIndexWriter)
> I don't _think_ this is a problem in most cases since the IndexReaders should all be readOnly in the core solr code) but plugins could attempt to use them in other ways. In general it seems like a really bad bug waiting to happen.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (SOLR-2654) not used consistently in
all places Directory objects are instantiated
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mark Miller updated SOLR-2654:
------------------------------
Fix Version/s: 4.0
> <lockType/> not used consistently in all places Directory objects are instantiated
> ----------------------------------------------------------------------------------
>
> Key: SOLR-2654
> URL: https://issues.apache.org/jira/browse/SOLR-2654
> Project: Solr
> Issue Type: Bug
> Reporter: Hoss Man
> Assignee: Mark Miller
> Priority: Critical
> Fix For: 3.4, 4.0
>
> Attachments: SOLR-2654.patch, SOLR-2654.patch
>
>
> nipunb noted on the mailing list then when configuring solr to use an alternate <lockType/> (ie: simple) the stats for the SolrIndexSearcher list NativeFSLockFactory being used by the Directory.
> The problem seems to be that SolrIndexConfig is not consulted when constructing Directory objects used for IndexReader (it's only used by SolrIndexWriter)
> I don't _think_ this is a problem in most cases since the IndexReaders should all be readOnly in the core solr code) but plugins could attempt to use them in other ways. In general it seems like a really bad bug waiting to happen.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2654) not used consistently in
all places Directory objects are instantiated
Posted by "Hoss Man (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13065608#comment-13065608 ]
Hoss Man commented on SOLR-2654:
--------------------------------
Original report...
{noformat}
From: nipunb
To: solr-user
Date: Thu, 14 Jul 2011 00:50:21 -0700 (PDT)
Subject: Lockfactory indicated by ../solr/admin/cores different from the setting in solrconfig.xml
hey
We have set locktype to simple in solrconfig.xml using:
<lockType>simple</lockType>. However on the page
localhost:8983/solr/admin/cores shows:
<str name="directory">
org.apache.lucene.store.MMapDirectory:org.apache.lucene.store.MMapDirectory..../solr-cores/core0/data/index
lockFactory=*org.apache.lucene.store.NativeFSLockFactory*@3415ddf5
</str>
I can see a write.lock file in the index directory. But shouldn't the page
solr/admin/cores also be showing the the lockFactory to be
org.apache.lucene.store.SimpleFSLockFactory.SimpleFSLockFactory
{noformat}
> <lockType/> not used consistently in all places Directory objects are instantiated
> ----------------------------------------------------------------------------------
>
> Key: SOLR-2654
> URL: https://issues.apache.org/jira/browse/SOLR-2654
> Project: Solr
> Issue Type: Bug
> Reporter: Hoss Man
> Priority: Critical
> Fix For: 3.4
>
>
> nipunb noted on the mailing list then when configuring solr to use an alternate <lockType/> (ie: simple) the stats for the SolrIndexSearcher list NativeFSLockFactory being used by the Directory.
> The problem seems to be that SolrIndexConfig is not consulted when constructing Directory objects used for IndexReader (it's only used by SolrIndexWriter)
> I don't _think_ this is a problem in most cases since the IndexReaders should all be readOnly in the core solr code) but plugins could attempt to use them in other ways. In general it seems like a really bad bug waiting to happen.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (SOLR-2654) not used consistently in
all places Directory objects are instantiated
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mark Miller updated SOLR-2654:
------------------------------
Attachment: SOLR-2654.patch
Another slightly better patch - we don't need to also track the set of all dirs anymore.
> <lockType/> not used consistently in all places Directory objects are instantiated
> ----------------------------------------------------------------------------------
>
> Key: SOLR-2654
> URL: https://issues.apache.org/jira/browse/SOLR-2654
> Project: Solr
> Issue Type: Bug
> Reporter: Hoss Man
> Assignee: Mark Miller
> Fix For: 3.4, 4.0
>
> Attachments: SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch, SOLR-2654.patch, SOLR-2698.patch
>
>
> nipunb noted on the mailing list then when configuring solr to use an alternate <lockType/> (ie: simple) the stats for the SolrIndexSearcher list NativeFSLockFactory being used by the Directory.
> The problem seems to be that SolrIndexConfig is not consulted when constructing Directory objects used for IndexReader (it's only used by SolrIndexWriter)
> I don't _think_ this is a problem in most cases since the IndexReaders should all be readOnly in the core solr code) but plugins could attempt to use them in other ways. In general it seems like a really bad bug waiting to happen.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org