You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Xin Li <xl...@book.com> on 2010/10/18 21:43:05 UTC
Spell checking question from a Solr novice
Hi,
I am looking for a quick solution to improve a search engine's spell checking performance. I was wondering if anyone tried to integrate Google SpellCheck API with Solr search engine (if possible). Google spellcheck came to my mind because of two reasons. First, it is costly to clean up the data to be used as spell check baseline. Secondly, google probably has the most complete set of misspelled search terms. That's why I would like to know if it is a feasible way to go.
Thanks,
Xin
This electronic mail message contains information that (a) is or
may be CONFIDENTIAL, PROPRIETARY IN NATURE, OR OTHERWISE
PROTECTED
BY LAW FROM DISCLOSURE, and (b) is intended only for the use of
the
addressee(s) named herein. If you are not an intended recipient,
please contact the sender immediately and take the steps
necessary
to delete the message completely from your computer system.
Not Intended as a Substitute for a Writing: Notwithstanding the
Uniform Electronic Transaction Act or any other law of similar
effect, absent an express statement to the contrary, this e-mail
message, its contents, and any attachments hereto are not
intended
to represent an offer or acceptance to enter into a contract and
are not otherwise intended to bind this sender,
barnesandnoble.com
llc, barnesandnoble.com inc. or any other person or entity.
Re: Spell checking question from a Solr novice
Posted by Pradeep Singh <pk...@gmail.com>.
I haven't yet but I was going to use the spell checker in the lucene contrib
module. That spellchecker is ngram based and previously I have noticed that
I get better results from ngram based spellcheck rather than fuzzy string
match based ones.
On Mon, Oct 18, 2010 at 12:43 PM, Xin Li <xl...@book.com> wrote:
> Hi,
>
> I am looking for a quick solution to improve a search engine's spell
> checking performance. I was wondering if anyone tried to integrate Google
> SpellCheck API with Solr search engine (if possible). Google spellcheck came
> to my mind because of two reasons. First, it is costly to clean up the data
> to be used as spell check baseline. Secondly, google probably has the most
> complete set of misspelled search terms. That's why I would like to know if
> it is a feasible way to go.
>
> Thanks,
> Xin
> This electronic mail message contains information that (a) is or
> may be CONFIDENTIAL, PROPRIETARY IN NATURE, OR OTHERWISE
> PROTECTED
> BY LAW FROM DISCLOSURE, and (b) is intended only for the use of
> the
> addressee(s) named herein. If you are not an intended recipient,
> please contact the sender immediately and take the steps
> necessary
> to delete the message completely from your computer system.
>
> Not Intended as a Substitute for a Writing: Notwithstanding the
> Uniform Electronic Transaction Act or any other law of similar
> effect, absent an express statement to the contrary, this e-mail
> message, its contents, and any attachments hereto are not
> intended
> to represent an offer or acceptance to enter into a contract and
> are not otherwise intended to bind this sender,
> barnesandnoble.com
> llc, barnesandnoble.com inc. or any other person or entity.
>
Re: Spell checking question from a Solr novice
Posted by Bill Dueber <bi...@dueber.com>.
On Mon, Oct 18, 2010 at 5:24 PM, Jason Blackerby <jb...@gmail.com>wrote:
> If you know the misspellings you could prevent them from being added to the
> dictionary with a StopFilterFactory like so:
>
Or, you know, correct the data :-)
--
Bill Dueber
Library Systems Programmer
University of Michigan Library
Re: Spell checking question from a Solr novice
Posted by Ezequiel Calderara <ez...@gmail.com>.
You can cross the new words against a dictionary and keep them in the file
as Jason described...
What Pradeep said is true, is always better to have "suggestions" related to
your index that have suggestions with no results...
On Mon, Oct 18, 2010 at 6:24 PM, Jason Blackerby <jb...@gmail.com>wrote:
> If you know the misspellings you could prevent them from being added to the
> dictionary with a StopFilterFactory like so:
>
> <fieldType name="textSpell" class="solr.TextField"
> positionIncrementGap="100" >
> <analyzer>
> <tokenizer class="solr.StandardTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> <filter class="solr.StopFilterFactory" ignoreCase="true"
> words="misspelled_words.txt"/>
> <filter class="solr.PatternReplaceFilterFactory" pattern="([^a-z])"
> replacement="" replace="all"/>
> <filter class="solr.LengthFilterFactory" min="2" max="50"/>
> </analyzer>
> </fieldType>
>
> where misspelled_words.txt contains the misspellings.
>
> On Mon, Oct 18, 2010 at 5:14 PM, Pradeep Singh <pk...@gmail.com>
> wrote:
>
> > I think a spellchecker based on your index has clear advantages. You can
> > spellcheck words specific to your domain which may not be available in an
> > outside dictionary. You can always dump the list from wordnet to get a
> > starter english dictionary.
> >
> > But then it also means that misspelled words from your domain become the
> > suggested correct word. Hmmm ... you'll need to have a way to prune out
> > such
> > words. Even then, your own domain based dictionary is a total go.
> >
> > On Mon, Oct 18, 2010 at 1:55 PM, Jonathan Rochkind <ro...@jhu.edu>
> > wrote:
> >
> > > In general, the benefit of the built-in Solr spellcheck is that it can
> > use
> > > a dictionary based on your actual index.
> > >
> > > If you want to use some external API, you certainly can, in your actual
> > > client app -- but it doesn't really need to involve Solr at all
> anymore,
> > > does it? Is there any benefit I'm not thinking of to doing that on the
> > solr
> > > side, instead of just in your client app?
> > >
> > > I think Yahoo (and maybe Microsoft?) have similar APIs with more
> generous
> > > ToSs, but I haven't looked in a while.
> > >
> > >
> > > Xin Li wrote:
> > >
> > >> Oops, never mind. Just read Google API policy. 1000 queries per day
> > limit
> > >> & for non-commercial use only.
> > >>
> > >>
> > >> -----Original Message-----
> > >> From: Xin Li Sent: Monday, October 18, 2010 3:43 PM
> > >> To: solr-user@lucene.apache.org
> > >> Subject: Spell checking question from a Solr novice
> > >>
> > >> Hi,
> > >> I am looking for a quick solution to improve a search engine's spell
> > >> checking performance. I was wondering if anyone tried to integrate
> > Google
> > >> SpellCheck API with Solr search engine (if possible). Google
> spellcheck
> > came
> > >> to my mind because of two reasons. First, it is costly to clean up the
> > data
> > >> to be used as spell check baseline. Secondly, google probably has the
> > most
> > >> complete set of misspelled search terms. That's why I would like to
> know
> > if
> > >> it is a feasible way to go.
> > >>
> > >> Thanks,
> > >> Xin
> > >> This electronic mail message contains information that (a) is or may
> be
> > >> CONFIDENTIAL, PROPRIETARY IN NATURE, OR OTHERWISE PROTECTED BY LAW
> FROM
> > >> DISCLOSURE, and (b) is intended only for the use of the
> > >> addressee(s) named herein. If you are not an intended recipient,
> please
> > >> contact the sender immediately and take the steps necessary to delete
> > the
> > >> message completely from your computer system.
> > >>
> > >> Not Intended as a Substitute for a Writing: Notwithstanding the
> Uniform
> > >> Electronic Transaction Act or any other law of similar effect, absent
> an
> > >> express statement to the contrary, this e-mail message, its contents,
> > and
> > >> any attachments hereto are not intended to represent an offer or
> > acceptance
> > >> to enter into a contract and are not otherwise intended to bind this
> > sender,
> > >> barnesandnoble.com llc, barnesandnoble.com inc. or any other person
> or
> > >> entity.
> > >> This electronic mail message contains information that (a) is or may
> be
> > >> CONFIDENTIAL, PROPRIETARY IN NATURE, OR OTHERWISE PROTECTED BY LAW
> FROM
> > >> DISCLOSURE, and (b) is intended only for the use of the
> > >> addressee(s) named herein. If you are not an intended recipient,
> please
> > >> contact the sender immediately and take the steps necessary to delete
> > the
> > >> message completely from your computer system.
> > >>
> > >> Not Intended as a Substitute for a Writing: Notwithstanding the
> Uniform
> > >> Electronic Transaction Act or any other law of similar effect, absent
> an
> > >> express statement to the contrary, this e-mail message, its contents,
> > and
> > >> any attachments hereto are not intended to represent an offer or
> > acceptance
> > >> to enter into a contract and are not otherwise intended to bind this
> > sender,
> > >> barnesandnoble.com llc, barnesandnoble.com inc. or any other person
> or
> > >> entity.
> > >>
> > >>
> > >
> >
>
--
______
Ezequiel.
Http://www.ironicnet.com
Re: Spell checking question from a Solr novice
Posted by Jason Blackerby <jb...@gmail.com>.
If you know the misspellings you could prevent them from being added to the
dictionary with a StopFilterFactory like so:
<fieldType name="textSpell" class="solr.TextField"
positionIncrementGap="100" >
<analyzer>
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.StopFilterFactory" ignoreCase="true"
words="misspelled_words.txt"/>
<filter class="solr.PatternReplaceFilterFactory" pattern="([^a-z])"
replacement="" replace="all"/>
<filter class="solr.LengthFilterFactory" min="2" max="50"/>
</analyzer>
</fieldType>
where misspelled_words.txt contains the misspellings.
On Mon, Oct 18, 2010 at 5:14 PM, Pradeep Singh <pk...@gmail.com> wrote:
> I think a spellchecker based on your index has clear advantages. You can
> spellcheck words specific to your domain which may not be available in an
> outside dictionary. You can always dump the list from wordnet to get a
> starter english dictionary.
>
> But then it also means that misspelled words from your domain become the
> suggested correct word. Hmmm ... you'll need to have a way to prune out
> such
> words. Even then, your own domain based dictionary is a total go.
>
> On Mon, Oct 18, 2010 at 1:55 PM, Jonathan Rochkind <ro...@jhu.edu>
> wrote:
>
> > In general, the benefit of the built-in Solr spellcheck is that it can
> use
> > a dictionary based on your actual index.
> >
> > If you want to use some external API, you certainly can, in your actual
> > client app -- but it doesn't really need to involve Solr at all anymore,
> > does it? Is there any benefit I'm not thinking of to doing that on the
> solr
> > side, instead of just in your client app?
> >
> > I think Yahoo (and maybe Microsoft?) have similar APIs with more generous
> > ToSs, but I haven't looked in a while.
> >
> >
> > Xin Li wrote:
> >
> >> Oops, never mind. Just read Google API policy. 1000 queries per day
> limit
> >> & for non-commercial use only.
> >>
> >>
> >> -----Original Message-----
> >> From: Xin Li Sent: Monday, October 18, 2010 3:43 PM
> >> To: solr-user@lucene.apache.org
> >> Subject: Spell checking question from a Solr novice
> >>
> >> Hi,
> >> I am looking for a quick solution to improve a search engine's spell
> >> checking performance. I was wondering if anyone tried to integrate
> Google
> >> SpellCheck API with Solr search engine (if possible). Google spellcheck
> came
> >> to my mind because of two reasons. First, it is costly to clean up the
> data
> >> to be used as spell check baseline. Secondly, google probably has the
> most
> >> complete set of misspelled search terms. That's why I would like to know
> if
> >> it is a feasible way to go.
> >>
> >> Thanks,
> >> Xin
> >> This electronic mail message contains information that (a) is or may be
> >> CONFIDENTIAL, PROPRIETARY IN NATURE, OR OTHERWISE PROTECTED BY LAW FROM
> >> DISCLOSURE, and (b) is intended only for the use of the
> >> addressee(s) named herein. If you are not an intended recipient, please
> >> contact the sender immediately and take the steps necessary to delete
> the
> >> message completely from your computer system.
> >>
> >> Not Intended as a Substitute for a Writing: Notwithstanding the Uniform
> >> Electronic Transaction Act or any other law of similar effect, absent an
> >> express statement to the contrary, this e-mail message, its contents,
> and
> >> any attachments hereto are not intended to represent an offer or
> acceptance
> >> to enter into a contract and are not otherwise intended to bind this
> sender,
> >> barnesandnoble.com llc, barnesandnoble.com inc. or any other person or
> >> entity.
> >> This electronic mail message contains information that (a) is or may be
> >> CONFIDENTIAL, PROPRIETARY IN NATURE, OR OTHERWISE PROTECTED BY LAW FROM
> >> DISCLOSURE, and (b) is intended only for the use of the
> >> addressee(s) named herein. If you are not an intended recipient, please
> >> contact the sender immediately and take the steps necessary to delete
> the
> >> message completely from your computer system.
> >>
> >> Not Intended as a Substitute for a Writing: Notwithstanding the Uniform
> >> Electronic Transaction Act or any other law of similar effect, absent an
> >> express statement to the contrary, this e-mail message, its contents,
> and
> >> any attachments hereto are not intended to represent an offer or
> acceptance
> >> to enter into a contract and are not otherwise intended to bind this
> sender,
> >> barnesandnoble.com llc, barnesandnoble.com inc. or any other person or
> >> entity.
> >>
> >>
> >
>
Re: Spell checking question from a Solr novice
Posted by Pradeep Singh <pk...@gmail.com>.
I think a spellchecker based on your index has clear advantages. You can
spellcheck words specific to your domain which may not be available in an
outside dictionary. You can always dump the list from wordnet to get a
starter english dictionary.
But then it also means that misspelled words from your domain become the
suggested correct word. Hmmm ... you'll need to have a way to prune out such
words. Even then, your own domain based dictionary is a total go.
On Mon, Oct 18, 2010 at 1:55 PM, Jonathan Rochkind <ro...@jhu.edu> wrote:
> In general, the benefit of the built-in Solr spellcheck is that it can use
> a dictionary based on your actual index.
>
> If you want to use some external API, you certainly can, in your actual
> client app -- but it doesn't really need to involve Solr at all anymore,
> does it? Is there any benefit I'm not thinking of to doing that on the solr
> side, instead of just in your client app?
>
> I think Yahoo (and maybe Microsoft?) have similar APIs with more generous
> ToSs, but I haven't looked in a while.
>
>
> Xin Li wrote:
>
>> Oops, never mind. Just read Google API policy. 1000 queries per day limit
>> & for non-commercial use only.
>>
>>
>> -----Original Message-----
>> From: Xin Li Sent: Monday, October 18, 2010 3:43 PM
>> To: solr-user@lucene.apache.org
>> Subject: Spell checking question from a Solr novice
>>
>> Hi,
>> I am looking for a quick solution to improve a search engine's spell
>> checking performance. I was wondering if anyone tried to integrate Google
>> SpellCheck API with Solr search engine (if possible). Google spellcheck came
>> to my mind because of two reasons. First, it is costly to clean up the data
>> to be used as spell check baseline. Secondly, google probably has the most
>> complete set of misspelled search terms. That's why I would like to know if
>> it is a feasible way to go.
>>
>> Thanks,
>> Xin
>> This electronic mail message contains information that (a) is or may be
>> CONFIDENTIAL, PROPRIETARY IN NATURE, OR OTHERWISE PROTECTED BY LAW FROM
>> DISCLOSURE, and (b) is intended only for the use of the
>> addressee(s) named herein. If you are not an intended recipient, please
>> contact the sender immediately and take the steps necessary to delete the
>> message completely from your computer system.
>>
>> Not Intended as a Substitute for a Writing: Notwithstanding the Uniform
>> Electronic Transaction Act or any other law of similar effect, absent an
>> express statement to the contrary, this e-mail message, its contents, and
>> any attachments hereto are not intended to represent an offer or acceptance
>> to enter into a contract and are not otherwise intended to bind this sender,
>> barnesandnoble.com llc, barnesandnoble.com inc. or any other person or
>> entity.
>> This electronic mail message contains information that (a) is or may be
>> CONFIDENTIAL, PROPRIETARY IN NATURE, OR OTHERWISE PROTECTED BY LAW FROM
>> DISCLOSURE, and (b) is intended only for the use of the
>> addressee(s) named herein. If you are not an intended recipient, please
>> contact the sender immediately and take the steps necessary to delete the
>> message completely from your computer system.
>>
>> Not Intended as a Substitute for a Writing: Notwithstanding the Uniform
>> Electronic Transaction Act or any other law of similar effect, absent an
>> express statement to the contrary, this e-mail message, its contents, and
>> any attachments hereto are not intended to represent an offer or acceptance
>> to enter into a contract and are not otherwise intended to bind this sender,
>> barnesandnoble.com llc, barnesandnoble.com inc. or any other person or
>> entity.
>>
>>
>
Re: Spell checking question from a Solr novice
Posted by Jonathan Rochkind <ro...@jhu.edu>.
In general, the benefit of the built-in Solr spellcheck is that it can
use a dictionary based on your actual index.
If you want to use some external API, you certainly can, in your actual
client app -- but it doesn't really need to involve Solr at all anymore,
does it? Is there any benefit I'm not thinking of to doing that on the
solr side, instead of just in your client app?
I think Yahoo (and maybe Microsoft?) have similar APIs with more
generous ToSs, but I haven't looked in a while.
Xin Li wrote:
> Oops, never mind. Just read Google API policy. 1000 queries per day limit & for non-commercial use only.
>
>
>
> -----Original Message-----
> From: Xin Li
> Sent: Monday, October 18, 2010 3:43 PM
> To: solr-user@lucene.apache.org
> Subject: Spell checking question from a Solr novice
>
> Hi,
>
> I am looking for a quick solution to improve a search engine's spell checking performance. I was wondering if anyone tried to integrate Google SpellCheck API with Solr search engine (if possible). Google spellcheck came to my mind because of two reasons. First, it is costly to clean up the data to be used as spell check baseline. Secondly, google probably has the most complete set of misspelled search terms. That's why I would like to know if it is a feasible way to go.
>
> Thanks,
> Xin
> This electronic mail message contains information that (a) is or
> may be CONFIDENTIAL, PROPRIETARY IN NATURE, OR OTHERWISE
> PROTECTED
> BY LAW FROM DISCLOSURE, and (b) is intended only for the use of
> the
> addressee(s) named herein. If you are not an intended recipient,
> please contact the sender immediately and take the steps
> necessary
> to delete the message completely from your computer system.
>
> Not Intended as a Substitute for a Writing: Notwithstanding the
> Uniform Electronic Transaction Act or any other law of similar
> effect, absent an express statement to the contrary, this e-mail
> message, its contents, and any attachments hereto are not
> intended
> to represent an offer or acceptance to enter into a contract and
> are not otherwise intended to bind this sender,
> barnesandnoble.com
> llc, barnesandnoble.com inc. or any other person or entity.
> This electronic mail message contains information that (a) is or
> may be CONFIDENTIAL, PROPRIETARY IN NATURE, OR OTHERWISE
> PROTECTED
> BY LAW FROM DISCLOSURE, and (b) is intended only for the use of
> the
> addressee(s) named herein. If you are not an intended recipient,
> please contact the sender immediately and take the steps
> necessary
> to delete the message completely from your computer system.
>
> Not Intended as a Substitute for a Writing: Notwithstanding the
> Uniform Electronic Transaction Act or any other law of similar
> effect, absent an express statement to the contrary, this e-mail
> message, its contents, and any attachments hereto are not
> intended
> to represent an offer or acceptance to enter into a contract and
> are not otherwise intended to bind this sender,
> barnesandnoble.com
> llc, barnesandnoble.com inc. or any other person or entity.
>
RE: Spell checking question from a Solr novice
Posted by Xin Li <xl...@book.com>.
Oops, never mind. Just read Google API policy. 1000 queries per day limit & for non-commercial use only.
-----Original Message-----
From: Xin Li
Sent: Monday, October 18, 2010 3:43 PM
To: solr-user@lucene.apache.org
Subject: Spell checking question from a Solr novice
Hi,
I am looking for a quick solution to improve a search engine's spell checking performance. I was wondering if anyone tried to integrate Google SpellCheck API with Solr search engine (if possible). Google spellcheck came to my mind because of two reasons. First, it is costly to clean up the data to be used as spell check baseline. Secondly, google probably has the most complete set of misspelled search terms. That's why I would like to know if it is a feasible way to go.
Thanks,
Xin
This electronic mail message contains information that (a) is or
may be CONFIDENTIAL, PROPRIETARY IN NATURE, OR OTHERWISE
PROTECTED
BY LAW FROM DISCLOSURE, and (b) is intended only for the use of
the
addressee(s) named herein. If you are not an intended recipient,
please contact the sender immediately and take the steps
necessary
to delete the message completely from your computer system.
Not Intended as a Substitute for a Writing: Notwithstanding the
Uniform Electronic Transaction Act or any other law of similar
effect, absent an express statement to the contrary, this e-mail
message, its contents, and any attachments hereto are not
intended
to represent an offer or acceptance to enter into a contract and
are not otherwise intended to bind this sender,
barnesandnoble.com
llc, barnesandnoble.com inc. or any other person or entity.
This electronic mail message contains information that (a) is or
may be CONFIDENTIAL, PROPRIETARY IN NATURE, OR OTHERWISE
PROTECTED
BY LAW FROM DISCLOSURE, and (b) is intended only for the use of
the
addressee(s) named herein. If you are not an intended recipient,
please contact the sender immediately and take the steps
necessary
to delete the message completely from your computer system.
Not Intended as a Substitute for a Writing: Notwithstanding the
Uniform Electronic Transaction Act or any other law of similar
effect, absent an express statement to the contrary, this e-mail
message, its contents, and any attachments hereto are not
intended
to represent an offer or acceptance to enter into a contract and
are not otherwise intended to bind this sender,
barnesandnoble.com
llc, barnesandnoble.com inc. or any other person or entity.
Re: Spell checking question from a Solr novice
Posted by Dennis Gearon <ge...@sbcglobal.net>.
The first question to ask is will it work for you.
The SECOND question is do you want google to know what's in your data?
Dennis Gearon
Signature Warning
----------------
It is always a good idea to learn from your own mistakes. It is usually a better idea to learn from others’ mistakes, so you do not have to make them yourself. from 'http://blogs.techrepublic.com.com/security/?p=4501&tag=nl.e036'
EARTH has a Right To Life,
otherwise we all die.
--- On Mon, 10/18/10, Xin Li <xl...@book.com> wrote:
> From: Xin Li <xl...@book.com>
> Subject: Spell checking question from a Solr novice
> To: solr-user@lucene.apache.org
> Date: Monday, October 18, 2010, 12:43 PM
> Hi,
>
> I am looking for a quick solution to improve a search
> engine's spell checking performance. I was wondering if
> anyone tried to integrate Google SpellCheck API with Solr
> search engine (if possible). Google spellcheck came to my
> mind because of two reasons. First, it is costly to clean up
> the data to be used as spell check baseline. Secondly,
> google probably has the most complete set of misspelled
> search terms. That's why I would like to know if it is a
> feasible way to go.
>
> Thanks,
> Xin
> This electronic mail message contains information that (a)
> is or
> may be CONFIDENTIAL, PROPRIETARY IN NATURE, OR OTHERWISE
> PROTECTED
> BY LAW FROM DISCLOSURE, and (b) is intended only for the
> use of
> the
> addressee(s) named herein. If you are not an intended
> recipient,
> please contact the sender immediately and take the steps
> necessary
> to delete the message completely from your computer
> system.
>
> Not Intended as a Substitute for a Writing: Notwithstanding
> the
> Uniform Electronic Transaction Act or any other law of
> similar
> effect, absent an express statement to the contrary, this
> e-mail
> message, its contents, and any attachments hereto are not
> intended
> to represent an offer or acceptance to enter into a
> contract and
> are not otherwise intended to bind this sender,
> barnesandnoble.com
> llc, barnesandnoble.com inc. or any other person or
> entity.
>