You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Mark Fenbers <ma...@noaa.gov> on 2015/09/18 23:34:47 UTC
write.lock
Greetings,
Whenever I try to build my spellcheck index
(params.set("spellcheck.build", true); or put a check in the
spellcheck.build box in the web interface) I get the following
stacktrace. Removing the write.lock file does no good. The message
comes right back anyway. I read in a post that increasing
writeLockTimeout would help. It did not help for me even increasing it
to 20,000 msec. If I don't build, then my resultset count is always 0,
i.e., empty results. What could be causing this?
Mark
Re: write.lock
Posted by Alessandro Benedetti <be...@gmail.com>.
Hi Mark,
let's summarise a little bit:
First of all you are using the IndexBasedSpellChecker which is what is
usually called "based on the sidecar index" .
Basically you are building a mini lucene index to be used with the
spellcheck component.
It behaves as a classic Lucene index, so it needs commits for visibility
and need to be updated with the terms coming from the main index.
>From the official doc :
*" The spellcheckIndexDir defines the location of the directory that holds
the spellcheck index, while the field defines the source field (defined
in schema.xml) for spell check terms. When choosing a field for the
spellcheck index, it's best to avoid a heavily processed field to get more
accurate results. If the field has many word variations from processing
synonyms and/or stemming, the dictionary will be created with those
variations in addition to more valid spelling data."*
If you don'[t want to keep a parallel sidecar index, you can go with the
DirectSpellcheck which works on the main index ( building an FSA in memory
for providing the misspelled suggestions).
>From the documentation :
" The DirectSolrSpellChecker uses terms from the Solr index without
building a parallel index like the IndexBasedSpellChecker. This spell
checker has the benefit of not having to be built regularly, meaning that
the terms are always up-to-date with terms in the index"
I am not so sure about this honestly, because I think it needs to be built
from time to time, even if the data structures are in memory.
I will double check this.
Related the FileBased one, will be similar to the IndexBased one, and will
produce a lucent index out of the original dictionary file.
I would honestly use 3 different directories, to avoid lock problems.
I would suggest to put them in the data directory ( sibling of the index
dir and tlog dir).
Or wherever you prefer.
Please give more details about the misspells you are not getting.
in the case remember that the max number of edits for the levenstein
distance has a well defined maximum ( it should be 2 edits for all the
spellcheck implementations).
Cheers
2015-09-22 14:19 GMT+01:00 Mark Fenbers <ma...@noaa.gov>:
> Mikhail,
>
> Yes, both the Index-based and File-based spell checkers reference the same
> index location. My understanding is they were supposed to. I didn't
> realize this was for writing indexes. Rather, I thought this was for
> reading the main index. So, I need to make 3 separate locations for
> indexes (main, index-based and File-based)?? Can I make them subdirs of the
> main index (in /localapps/dev/EventLog/index)? Or would that mess up the
> main index?
>
> Thanks for raising my awareness of these errors!
> Mark
>
>
> On 9/21/2015 5:07 PM, Mikhail Khludnev wrote:
>
>> Both of these guys below try to write spell index into the same dir.
>> Don't they?
>>
>> To make it clear, it's not possible so far.
>>
>> <lst name="spellchecker">
>> <str name="classname">solr.IndexBasedSpellChecker</str>
>> <str name="spellcheckIndexDir">/localapps/dev/EventLog/index</str>
>>
>>
>> <lst name="spellchecker">
>> <str name="classname">solr.FileBasedSpellChecker</str>
>> <str
>> name="spellcheckIndexDir">/localapps/dev/EventLog/index</str>
>>
>> Also, can you make sure that this path doesn't lead to main index dir.
>>
>>
>> On Mon, Sep 21, 2015 at 5:13 PM, Mark Fenbers <ma...@noaa.gov>
>> wrote:
>>
>>
>>
>
--
--------------------------
Benedetti Alessandro
Visiting card - http://about.me/alessandro_benedetti
Blog - http://alexbenedetti.blogspot.co.uk
"Tyger, tyger burning bright
In the forests of the night,
What immortal hand or eye
Could frame thy fearful symmetry?"
William Blake - Songs of Experience -1794 England
Re: write.lock
Posted by Mark Fenbers <ma...@noaa.gov>.
Mikhail,
Yes, both the Index-based and File-based spell checkers reference the
same index location. My understanding is they were supposed to. I
didn't realize this was for writing indexes. Rather, I thought this was
for reading the main index. So, I need to make 3 separate locations for
indexes (main, index-based and File-based)?? Can I make them subdirs of
the main index (in /localapps/dev/EventLog/index)? Or would that mess
up the main index?
Thanks for raising my awareness of these errors!
Mark
On 9/21/2015 5:07 PM, Mikhail Khludnev wrote:
> Both of these guys below try to write spell index into the same dir. Don't they?
>
> To make it clear, it's not possible so far.
>
> <lst name="spellchecker">
> <str name="classname">solr.IndexBasedSpellChecker</str>
> <str name="spellcheckIndexDir">/localapps/dev/EventLog/index</str>
>
>
> <lst name="spellchecker">
> <str name="classname">solr.FileBasedSpellChecker</str>
> <str name="spellcheckIndexDir">/localapps/dev/EventLog/index</str>
>
> Also, can you make sure that this path doesn't lead to main index dir.
>
>
> On Mon, Sep 21, 2015 at 5:13 PM, Mark Fenbers <ma...@noaa.gov> wrote:
>
>
Re: write.lock
Posted by Mark Fenbers <ma...@noaa.gov>.
OK, I gave each of these spellcheckIndexDir tokens distinct location --
from each other and from the main index. This has resolved the
write.lock problem when I attempt a spellcheck.build! Thanks for the help!
I looked in the new spellcheckIndexDir location and the directory is
populated with a few files. So it seems "spellcheck.build" worked, but
I am still not getting any hits when I purposefully misspell a word.
But I'll post this problem with more details in a separate post.
Mark
On 9/21/2015 5:07 PM, Mikhail Khludnev wrote:
> Both of these guys below try to write spell index into the same dir. Don't they?
>
> To make it clear, it's not possible so far.
>
> <lst name="spellchecker">
> <str name="classname">solr.IndexBasedSpellChecker</str>
> <str name="spellcheckIndexDir">/localapps/dev/EventLog/index</str>
>
>
> <lst name="spellchecker">
> <str name="classname">solr.FileBasedSpellChecker</str>
> <str name="spellcheckIndexDir">/localapps/dev/EventLog/index</str>
>
> Also, can you make sure that this path doesn't lead to main index dir.
>
>
> On Mon, Sep 21, 2015 at 5:13 PM, Mark Fenbers <ma...@noaa.gov> wrote:
>
>> A snippet of my solrconfig.xml is attached. The snippet only contains the
>> Spell checking sections (for brevity) which should be sufficient for you to
>> see all the pertinent info you seek.
>>
>> Thanks!
>> Mark
>>
>>
>> On 9/19/2015 3:29 AM, Mikhail Khludnev wrote:
>>
>>> Mark,
>>>
>>> What's your solconfig.xml?
>>>
>>> On Sat, Sep 19, 2015 at 12:34 AM, Mark Fenbers <ma...@noaa.gov>
>>> wrote:
>>>
>>> Greetings,
>>>> Whenever I try to build my spellcheck index
>>>> (params.set("spellcheck.build", true); or put a check in the
>>>> spellcheck.build box in the web interface) I get the following
>>>> stacktrace.
>>>> Removing the write.lock file does no good. The message comes right back
>>>> anyway. I read in a post that increasing writeLockTimeout would help.
>>>> It
>>>> did not help for me even increasing it to 20,000 msec. If I don't build,
>>>> then my resultset count is always 0, i.e., empty results. What could be
>>>> causing this?
>>>>
>>>> Mark
>>>>
>>>>
>>>
>
Re: write.lock
Posted by Mikhail Khludnev <mk...@griddynamics.com>.
Both of these guys below try to write spell index into the same dir. Don't they?
To make it clear, it's not possible so far.
<lst name="spellchecker">
<str name="classname">solr.IndexBasedSpellChecker</str>
<str name="spellcheckIndexDir">/localapps/dev/EventLog/index</str>
<lst name="spellchecker">
<str name="classname">solr.FileBasedSpellChecker</str>
<str name="spellcheckIndexDir">/localapps/dev/EventLog/index</str>
Also, can you make sure that this path doesn't lead to main index dir.
On Mon, Sep 21, 2015 at 5:13 PM, Mark Fenbers <ma...@noaa.gov> wrote:
> A snippet of my solrconfig.xml is attached. The snippet only contains the
> Spell checking sections (for brevity) which should be sufficient for you to
> see all the pertinent info you seek.
>
> Thanks!
> Mark
>
>
> On 9/19/2015 3:29 AM, Mikhail Khludnev wrote:
>
>> Mark,
>>
>> What's your solconfig.xml?
>>
>> On Sat, Sep 19, 2015 at 12:34 AM, Mark Fenbers <ma...@noaa.gov>
>> wrote:
>>
>> Greetings,
>>>
>>> Whenever I try to build my spellcheck index
>>> (params.set("spellcheck.build", true); or put a check in the
>>> spellcheck.build box in the web interface) I get the following
>>> stacktrace.
>>> Removing the write.lock file does no good. The message comes right back
>>> anyway. I read in a post that increasing writeLockTimeout would help.
>>> It
>>> did not help for me even increasing it to 20,000 msec. If I don't build,
>>> then my resultset count is always 0, i.e., empty results. What could be
>>> causing this?
>>>
>>> Mark
>>>
>>>
>>
>>
>
--
Sincerely yours
Mikhail Khludnev
Principal Engineer,
Grid Dynamics
<http://www.griddynamics.com>
<mk...@griddynamics.com>
Re: write.lock
Posted by Mark Fenbers <ma...@noaa.gov>.
A snippet of my solrconfig.xml is attached. The snippet only contains
the Spell checking sections (for brevity) which should be sufficient for
you to see all the pertinent info you seek.
Thanks!
Mark
On 9/19/2015 3:29 AM, Mikhail Khludnev wrote:
> Mark,
>
> What's your solconfig.xml?
>
> On Sat, Sep 19, 2015 at 12:34 AM, Mark Fenbers <ma...@noaa.gov>
> wrote:
>
>> Greetings,
>>
>> Whenever I try to build my spellcheck index
>> (params.set("spellcheck.build", true); or put a check in the
>> spellcheck.build box in the web interface) I get the following stacktrace.
>> Removing the write.lock file does no good. The message comes right back
>> anyway. I read in a post that increasing writeLockTimeout would help. It
>> did not help for me even increasing it to 20,000 msec. If I don't build,
>> then my resultset count is always 0, i.e., empty results. What could be
>> causing this?
>>
>> Mark
>>
>
>
Re: write.lock
Posted by Mikhail Khludnev <mk...@griddynamics.com>.
Mark,
What's your solconfig.xml?
On Sat, Sep 19, 2015 at 12:34 AM, Mark Fenbers <ma...@noaa.gov>
wrote:
> Greetings,
>
> Whenever I try to build my spellcheck index
> (params.set("spellcheck.build", true); or put a check in the
> spellcheck.build box in the web interface) I get the following stacktrace.
> Removing the write.lock file does no good. The message comes right back
> anyway. I read in a post that increasing writeLockTimeout would help. It
> did not help for me even increasing it to 20,000 msec. If I don't build,
> then my resultset count is always 0, i.e., empty results. What could be
> causing this?
>
> Mark
>
--
Sincerely yours
Mikhail Khludnev
Principal Engineer,
Grid Dynamics
<http://www.griddynamics.com>
<mk...@griddynamics.com>