You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Mark Fenbers <ma...@noaa.gov> on 2015/10/12 21:37:56 UTC
File-based Spelling
Greetings!
I'm attempting to use a file-based spell checker. My sourceLocation is
/usr/share/dict/linux.words, and my spellcheckIndexDir is set to
./data/spFile. BuildOnStartup is set to true, and I see nothing to
suggest any sort of problem/error in solr.log. However, in my
./data/spFile/ directory, there are only two files: segments_2 with only
71 bytes in it, and a zero-byte write.lock file. For a source
dictionary having 480,000 words in it, I was expecting a bit more
substance in the ./data/spFile directory. Something doesn't seem right
with this.
Moreover, I ran a query on the word Fenbers, which isn't listed in the
linux.words file, but there are several similar words. The results I
got back were odd, and suggestions included the following:
fenber
f en be r
f e nb er
f en b er
f e n be r
f en b e r
f e nb e r
f e n b er
f e n b e r
But I expected suggestions like fenders, embers, and fenberry, etc. I
also ran a query on Mark (which IS listed in linux.words) and got back
two suggestions in a similar format. I played with configurables like
changing the fieldType from text_en to string and the characterEncoding
from UTF-8 to ASCII, etc., but nothing seemed to yield any different
results.
Can anyone offer suggestions as to what I'm doing wrong? I've been
struggling with this for more than 40 hours now! I'm surprised my
persistence has lasted this long!
Thanks,
Mark
Re: File-based Spelling
Posted by Erick Erickson <er...@gmail.com>.
Let's see your solrconfig entries? Doubtless something innocent
seeming isn't quite right.
This might provide some clues:
http://lucidworks.com/blog/2015/03/04/solr-suggester/
The reference guide is the first place to look, a lot of this
functionality has changed
in recent years so I always try to use the Solr reference guide:
https://cwiki.apache.org/confluence/display/solr/Spell+Checking
Best,
Erick
On Mon, Oct 12, 2015 at 12:37 PM, Mark Fenbers <ma...@noaa.gov> wrote:
> Greetings!
>
> I'm attempting to use a file-based spell checker. My sourceLocation is
> /usr/share/dict/linux.words, and my spellcheckIndexDir is set to
> ./data/spFile. BuildOnStartup is set to true, and I see nothing to suggest
> any sort of problem/error in solr.log. However, in my ./data/spFile/
> directory, there are only two files: segments_2 with only 71 bytes in it,
> and a zero-byte write.lock file. For a source dictionary having 480,000
> words in it, I was expecting a bit more substance in the ./data/spFile
> directory. Something doesn't seem right with this.
>
> Moreover, I ran a query on the word Fenbers, which isn't listed in the
> linux.words file, but there are several similar words. The results I got
> back were odd, and suggestions included the following:
> fenber
> f en be r
> f e nb er
> f en b er
> f e n be r
> f en b e r
> f e nb e r
> f e n b er
> f e n b e r
>
> But I expected suggestions like fenders, embers, and fenberry, etc. I also
> ran a query on Mark (which IS listed in linux.words) and got back two
> suggestions in a similar format. I played with configurables like changing
> the fieldType from text_en to string and the characterEncoding from UTF-8 to
> ASCII, etc., but nothing seemed to yield any different results.
>
> Can anyone offer suggestions as to what I'm doing wrong? I've been
> struggling with this for more than 40 hours now! I'm surprised my
> persistence has lasted this long!
>
> Thanks,
> Mark
Re: File-based Spelling
Posted by Mark Fenbers <ma...@noaa.gov>.
OK. I removed it, started Solr, adn refreshed the query, but my results
are the same, indicating that queryAnalyzerFieldType has nothing to do
with my problem.
New ideas??
Mark
On 10/19/2015 4:37 AM, Duck Geraint (ext) GBJH wrote:
> "Yet, it claimed it found my misspelled word to be "fenber" without the "s""
> I wonder if this is because you seem to applying a stemmer to your dictionary words.
>
> Try removing the "<str name="queryAnalyzerFieldType">text_en</str>" line from your spellcheck search component definition.
>
> Geraint
>
>
> Geraint Duck
> Data Scientist
> Toxicology and Health Sciences
> Syngenta UK
> Email: geraint.duck@syngenta.com
>
>
> -----Original Message-----
> From: Mark Fenbers [mailto:mark.fenbers@noaa.gov]
> Sent: 16 October 2015 19:43
> To: solr-user@lucene.apache.org
> Subject: Re: File-based Spelling
>
> On 10/13/2015 9:30 AM, Dyer, James wrote:
>> Mark,
>>
>> The older spellcheck implementations create an n-gram sidecar index, which is why you're seeing your name split into 2-grams like this. See the IR Book by Manning et al, section 3.3.4 for more information. Based on the results you're getting, I think it is loading your file correctly. You should now try a query against this spelling index, using words *not* in the file you loaded that are within 1 or 2 edits from something that is in the dictionary. If it doesn't yield suggestions, then post the relevant sections of the solrconfig.xml, schema.xml and also the query string you are trying.
>>
>> James Dyer
>> Ingram Content Group
>>
> James, I've already done this. My query string was "fenbers". This is
> my last name which does *not* occur in the linux.words file. It is only
> 1 edit distance from "fenders" which *is* in the linux.words file. Yet, it claimed it found my misspelled word to be "fenber" without the "s"
> and it gave me these 8 suggestions:
> f en be r
> f e nb er
> f en b er
> f e n be r
> f en b e r
> f e nb e r
> f e n b er
> f e n b e r
>
> So I'm attaching the the entire solrconfig.xml and schema.xml that is in effect. These are in a single file with all the block comments removed.
>
> I'm also puzzled that you say "older implementations create a sidecar index"... because I am using v5.3.0, which was the latest version as of my download a month or two ago. So, with my implementation being recent, why is an n-gram sidecar index still (seemingly) being produced?
>
> thanks for the help!
> Mark
>
>
>
> ________________________________
>
>
> Syngenta Limited, Registered in England No 2710846;Registered Office : Syngenta Limited, European Regional Centre, Priestley Road, Surrey Research Park, Guildford, Surrey, GU2 7YH, United Kingdom
> ________________________________
> This message may contain confidential information. If you are not the designated recipient, please notify the sender immediately, and delete the original and any copies. Any use of the message by you is prohibited.
RE: File-based Spelling
Posted by "Duck Geraint (ext) GBJH" <ge...@syngenta.com>.
"Yet, it claimed it found my misspelled word to be "fenber" without the "s""
I wonder if this is because you seem to applying a stemmer to your dictionary words.
Try removing the "<str name="queryAnalyzerFieldType">text_en</str>" line from your spellcheck search component definition.
Geraint
Geraint Duck
Data Scientist
Toxicology and Health Sciences
Syngenta UK
Email: geraint.duck@syngenta.com
-----Original Message-----
From: Mark Fenbers [mailto:mark.fenbers@noaa.gov]
Sent: 16 October 2015 19:43
To: solr-user@lucene.apache.org
Subject: Re: File-based Spelling
On 10/13/2015 9:30 AM, Dyer, James wrote:
> Mark,
>
> The older spellcheck implementations create an n-gram sidecar index, which is why you're seeing your name split into 2-grams like this. See the IR Book by Manning et al, section 3.3.4 for more information. Based on the results you're getting, I think it is loading your file correctly. You should now try a query against this spelling index, using words *not* in the file you loaded that are within 1 or 2 edits from something that is in the dictionary. If it doesn't yield suggestions, then post the relevant sections of the solrconfig.xml, schema.xml and also the query string you are trying.
>
> James Dyer
> Ingram Content Group
>
James, I've already done this. My query string was "fenbers". This is
my last name which does *not* occur in the linux.words file. It is only
1 edit distance from "fenders" which *is* in the linux.words file. Yet, it claimed it found my misspelled word to be "fenber" without the "s"
and it gave me these 8 suggestions:
f en be r
f e nb er
f en b er
f e n be r
f en b e r
f e nb e r
f e n b er
f e n b e r
So I'm attaching the the entire solrconfig.xml and schema.xml that is in effect. These are in a single file with all the block comments removed.
I'm also puzzled that you say "older implementations create a sidecar index"... because I am using v5.3.0, which was the latest version as of my download a month or two ago. So, with my implementation being recent, why is an n-gram sidecar index still (seemingly) being produced?
thanks for the help!
Mark
________________________________
Syngenta Limited, Registered in England No 2710846;Registered Office : Syngenta Limited, European Regional Centre, Priestley Road, Surrey Research Park, Guildford, Surrey, GU2 7YH, United Kingdom
________________________________
This message may contain confidential information. If you are not the designated recipient, please notify the sender immediately, and delete the original and any copies. Any use of the message by you is prohibited.
Re: File-based Spelling
Posted by Mark Fenbers <ma...@noaa.gov>.
On 10/13/2015 9:30 AM, Dyer, James wrote:
> Mark,
>
> The older spellcheck implementations create an n-gram sidecar index, which is why you're seeing your name split into 2-grams like this. See the IR Book by Manning et al, section 3.3.4 for more information. Based on the results you're getting, I think it is loading your file correctly. You should now try a query against this spelling index, using words *not* in the file you loaded that are within 1 or 2 edits from something that is in the dictionary. If it doesn't yield suggestions, then post the relevant sections of the solrconfig.xml, schema.xml and also the query string you are trying.
>
> James Dyer
> Ingram Content Group
>
James, I've already done this. My query string was "fenbers". This is
my last name which does *not* occur in the linux.words file. It is only
1 edit distance from "fenders" which *is* in the linux.words file. Yet,
it claimed it found my misspelled word to be "fenber" without the "s"
and it gave me these 8 suggestions:
f en be r
f e nb er
f en b er
f e n be r
f en b e r
f e nb e r
f e n b er
f e n b e r
So I'm attaching the the entire solrconfig.xml and schema.xml that is in
effect. These are in a single file with all the block comments removed.
I'm also puzzled that you say "older implementations create a sidecar
index"... because I am using v5.3.0, which was the latest version as of
my download a month or two ago. So, with my implementation being
recent, why is an n-gram sidecar index still (seemingly) being produced?
thanks for the help!
Mark
RE: File-based Spelling
Posted by "Dyer, James" <Ja...@ingramcontent.com>.
Mark,
The older spellcheck implementations create an n-gram sidecar index, which is why you're seeing your name split into 2-grams like this. See the IR Book by Manning et al, section 3.3.4 for more information. Based on the results you're getting, I think it is loading your file correctly. You should now try a query against this spelling index, using words *not* in the file you loaded that are within 1 or 2 edits from something that is in the dictionary. If it doesn't yield suggestions, then post the relevant sections of the solrconfig.xml, schema.xml and also the query string you are trying.
James Dyer
Ingram Content Group
-----Original Message-----
From: Mark Fenbers [mailto:mark.fenbers@noaa.gov]
Sent: Monday, October 12, 2015 2:38 PM
To: Solr User Group
Subject: File-based Spelling
Greetings!
I'm attempting to use a file-based spell checker. My sourceLocation is
/usr/share/dict/linux.words, and my spellcheckIndexDir is set to
./data/spFile. BuildOnStartup is set to true, and I see nothing to
suggest any sort of problem/error in solr.log. However, in my
./data/spFile/ directory, there are only two files: segments_2 with only
71 bytes in it, and a zero-byte write.lock file. For a source
dictionary having 480,000 words in it, I was expecting a bit more
substance in the ./data/spFile directory. Something doesn't seem right
with this.
Moreover, I ran a query on the word Fenbers, which isn't listed in the
linux.words file, but there are several similar words. The results I
got back were odd, and suggestions included the following:
fenber
f en be r
f e nb er
f en b er
f e n be r
f en b e r
f e nb e r
f e n b er
f e n b e r
But I expected suggestions like fenders, embers, and fenberry, etc. I
also ran a query on Mark (which IS listed in linux.words) and got back
two suggestions in a similar format. I played with configurables like
changing the fieldType from text_en to string and the characterEncoding
from UTF-8 to ASCII, etc., but nothing seemed to yield any different
results.
Can anyone offer suggestions as to what I'm doing wrong? I've been
struggling with this for more than 40 hours now! I'm surprised my
persistence has lasted this long!
Thanks,
Mark