You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Clemens Wyss <cl...@mysign.ch> on 2011/05/05 14:54:21 UTC
Using Solr's (Auto)suggest with plain lucene
I have implemented my index (in fact it's a plugable indexing API) in "plain Lucene". It tried to implement a term suggestion mechanism on my own, being not to happy so far.
At
http://search-lucene.com/m/0QBv41ssGlh/suggestion&subj=Auto+Suggest
I have seen Solr's auto suggestion for search terms. Appart from not being "fuzzy" it looks quite promising.
How easily can I Integrate Solr's suggestion with my Lucene index?
Thx
Clemens
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Using Solr's (Auto)suggest with plain lucene
Posted by Dawid Weiss <da...@gmail.com>.
FSTLookup is strictly prefix-based. JaspellLookup should handle the fuzzy
matching, but I know very little about it. The source code is your friend,
though -- just try what works and what doesn't.
Dawid
On Fri, May 6, 2011 at 3:21 PM, Clemens Wyss <cl...@mysign.ch> wrote:
> What if I'd like to add "fuzzy" lookup for terms?
>
> E.g.:
> "melo" should also bring up "merlot"
>
> > -----Ursprüngliche Nachricht-----
> > Von: Dawid Weiss [mailto:dawid.weiss@gmail.com]
> > Gesendet: Freitag, 6. Mai 2011 11:30
> > An: java-user@lucene.apache.org
> > Betreff: Re: Using Solr's (Auto)suggest with plain lucene
> >
> > So you'd like case-sensitive lookup, not case-insensitive lookup.
> FSTLookup
> > just takes characters from the input and does not perform any
> normalization.
> > So, if you feed it terms with casing, you'll get (cased) suggestions
> back.
> > If you need cased suggestions, but provide normalized (lowercased)
> prefixes
> > you'll get nothing, although such a feature would be relatively easy to
> > implement based on the automaton code currently in the SVN.
> >
> > Dawid
> >
> > On Fri, May 6, 2011 at 11:18 AM, Clemens Wyss <cl...@mysign.ch>
> > wrote:
> >
> > > Thx Dawid,
> > > Could you be more precise on 2. ? I'd like to leave the suggested
> > > terms untouched, i.e. cased.
> > >
> > > Clemens
> > >
> > > > -----Ursprüngliche Nachricht-----
> > > > Von: Dawid Weiss [mailto:dawid.weiss@gmail.com]
> > > > Gesendet: Freitag, 6. Mai 2011 11:12
> > > > An: java-user@lucene.apache.org
> > > > Betreff: Re: Using Solr's (Auto)suggest with plain lucene
> > > >
> > > > 1. FSTLookup is in the trunk only, it has not been released.
> > > > 2. Case-insensitive lookups are possible, you'd just need to pass
> > > normalized
> > > > tokens from which the dictionary is built and then use the same
> > > > normalization at the time you query for suggestions.
> > > > 3. "http://search-lucene.com/m/586gA4ccL11". I have no idea.
> > > >
> > > > Dawid
> > > >
> > > > On Fri, May 6, 2011 at 11:06 AM, Clemens Wyss
> > <cl...@mysign.ch>
> > > > wrote:
> > > >
> > > > > I have come across TSTLookup.
> > > > > In which jar Do I find FSTLookup? Does any of these lookup-classes
> > > > > allow case insensitive lookups? My terms are (unfortunately) cased.
> > > > >
> > > > > What Lookup "sits" behind this suggester
> > > > > http://search-lucene.com/m/586gA4ccL11
> > > > > here?
> > > > >
> > > > >
> > > > > > -----Ursprüngliche Nachricht-----
> > > > > > Von: Dawid Weiss [mailto:dawid.weiss@gmail.com]
> > > > > > Gesendet: Donnerstag, 5. Mai 2011 15:00
> > > > > > An: java-user@lucene.apache.org
> > > > > > Betreff: Re: Using Solr's (Auto)suggest with plain lucene
> > > > > >
> > > > > > If you check out the source code of solr/lucene, look at
> > > > > > FSTLookup class
> > > > > and
> > > > > > FSTLookupTest -- you can populate FSTLookup manually with terms/
> > > > > > phrases from your index and then use the resulting automaton for
> > > > suggestions.
> > > > > >
> > > > > > Dawid
> > > > > >
> > > > > > On Thu, May 5, 2011 at 2:54 PM, Clemens Wyss
> > > > <cl...@mysign.ch>
> > > > > > wrote:
> > > > > >
> > > > > > > I have implemented my index (in fact it's a plugable indexing
> > > > > > > API) in "plain Lucene". It tried to implement a term
> > > > > > > suggestion mechanism on my own, being not to happy so far.
> > > > > > > At
> > > > > > > http://search-
> > > > > > lucene.com/m/0QBv41ssGlh/suggestion&subj=Auto+Suggest
> > > > > > > I have seen Solr's auto suggestion for search terms. Appart
> > > > > > > from not being "fuzzy" it looks quite promising.
> > > > > > >
> > > > > > > How easily can I Integrate Solr's suggestion with my Lucene
> index?
> > > > > > >
> > > > > > > Thx
> > > > > > > Clemens
> > > > > > >
> > > > > > > --------------------------------------------------------------
> > > > > > > ----
> > > > > > > --- To unsubscribe, e-mail:
> > > > > > > java-user-unsubscribe@lucene.apache.org
> > > > > > > For additional commands, e-mail:
> > > > > > > java-user-help@lucene.apache.org
> > > > > > >
> > > > > > >
> > > > >
> > >
>
AW: Using Solr's (Auto)suggest with plain lucene
Posted by Clemens Wyss <cl...@mysign.ch>.
What if I'd like to add "fuzzy" lookup for terms?
E.g.:
"melo" should also bring up "merlot"
> -----Ursprüngliche Nachricht-----
> Von: Dawid Weiss [mailto:dawid.weiss@gmail.com]
> Gesendet: Freitag, 6. Mai 2011 11:30
> An: java-user@lucene.apache.org
> Betreff: Re: Using Solr's (Auto)suggest with plain lucene
>
> So you'd like case-sensitive lookup, not case-insensitive lookup. FSTLookup
> just takes characters from the input and does not perform any normalization.
> So, if you feed it terms with casing, you'll get (cased) suggestions back.
> If you need cased suggestions, but provide normalized (lowercased) prefixes
> you'll get nothing, although such a feature would be relatively easy to
> implement based on the automaton code currently in the SVN.
>
> Dawid
>
> On Fri, May 6, 2011 at 11:18 AM, Clemens Wyss <cl...@mysign.ch>
> wrote:
>
> > Thx Dawid,
> > Could you be more precise on 2. ? I'd like to leave the suggested
> > terms untouched, i.e. cased.
> >
> > Clemens
> >
> > > -----Ursprüngliche Nachricht-----
> > > Von: Dawid Weiss [mailto:dawid.weiss@gmail.com]
> > > Gesendet: Freitag, 6. Mai 2011 11:12
> > > An: java-user@lucene.apache.org
> > > Betreff: Re: Using Solr's (Auto)suggest with plain lucene
> > >
> > > 1. FSTLookup is in the trunk only, it has not been released.
> > > 2. Case-insensitive lookups are possible, you'd just need to pass
> > normalized
> > > tokens from which the dictionary is built and then use the same
> > > normalization at the time you query for suggestions.
> > > 3. "http://search-lucene.com/m/586gA4ccL11". I have no idea.
> > >
> > > Dawid
> > >
> > > On Fri, May 6, 2011 at 11:06 AM, Clemens Wyss
> <cl...@mysign.ch>
> > > wrote:
> > >
> > > > I have come across TSTLookup.
> > > > In which jar Do I find FSTLookup? Does any of these lookup-classes
> > > > allow case insensitive lookups? My terms are (unfortunately) cased.
> > > >
> > > > What Lookup "sits" behind this suggester
> > > > http://search-lucene.com/m/586gA4ccL11
> > > > here?
> > > >
> > > >
> > > > > -----Ursprüngliche Nachricht-----
> > > > > Von: Dawid Weiss [mailto:dawid.weiss@gmail.com]
> > > > > Gesendet: Donnerstag, 5. Mai 2011 15:00
> > > > > An: java-user@lucene.apache.org
> > > > > Betreff: Re: Using Solr's (Auto)suggest with plain lucene
> > > > >
> > > > > If you check out the source code of solr/lucene, look at
> > > > > FSTLookup class
> > > > and
> > > > > FSTLookupTest -- you can populate FSTLookup manually with terms/
> > > > > phrases from your index and then use the resulting automaton for
> > > suggestions.
> > > > >
> > > > > Dawid
> > > > >
> > > > > On Thu, May 5, 2011 at 2:54 PM, Clemens Wyss
> > > <cl...@mysign.ch>
> > > > > wrote:
> > > > >
> > > > > > I have implemented my index (in fact it's a plugable indexing
> > > > > > API) in "plain Lucene". It tried to implement a term
> > > > > > suggestion mechanism on my own, being not to happy so far.
> > > > > > At
> > > > > > http://search-
> > > > > lucene.com/m/0QBv41ssGlh/suggestion&subj=Auto+Suggest
> > > > > > I have seen Solr's auto suggestion for search terms. Appart
> > > > > > from not being "fuzzy" it looks quite promising.
> > > > > >
> > > > > > How easily can I Integrate Solr's suggestion with my Lucene index?
> > > > > >
> > > > > > Thx
> > > > > > Clemens
> > > > > >
> > > > > > --------------------------------------------------------------
> > > > > > ----
> > > > > > --- To unsubscribe, e-mail:
> > > > > > java-user-unsubscribe@lucene.apache.org
> > > > > > For additional commands, e-mail:
> > > > > > java-user-help@lucene.apache.org
> > > > > >
> > > > > >
> > > >
> >
Re: Using Solr's (Auto)suggest with plain lucene
Posted by Dawid Weiss <da...@gmail.com>.
So you'd like case-sensitive lookup, not case-insensitive lookup. FSTLookup
just takes characters from the input and does not perform any normalization.
So, if you feed it terms with casing, you'll get (cased) suggestions back.
If you need cased suggestions, but provide normalized (lowercased) prefixes
you'll get nothing, although such a feature would be relatively easy to
implement based on the automaton code currently in the SVN.
Dawid
On Fri, May 6, 2011 at 11:18 AM, Clemens Wyss <cl...@mysign.ch> wrote:
> Thx Dawid,
> Could you be more precise on 2. ? I'd like to leave the suggested terms
> untouched, i.e. cased.
>
> Clemens
>
> > -----Ursprüngliche Nachricht-----
> > Von: Dawid Weiss [mailto:dawid.weiss@gmail.com]
> > Gesendet: Freitag, 6. Mai 2011 11:12
> > An: java-user@lucene.apache.org
> > Betreff: Re: Using Solr's (Auto)suggest with plain lucene
> >
> > 1. FSTLookup is in the trunk only, it has not been released.
> > 2. Case-insensitive lookups are possible, you'd just need to pass
> normalized
> > tokens from which the dictionary is built and then use the same
> > normalization at the time you query for suggestions.
> > 3. "http://search-lucene.com/m/586gA4ccL11". I have no idea.
> >
> > Dawid
> >
> > On Fri, May 6, 2011 at 11:06 AM, Clemens Wyss <cl...@mysign.ch>
> > wrote:
> >
> > > I have come across TSTLookup.
> > > In which jar Do I find FSTLookup? Does any of these lookup-classes
> > > allow case insensitive lookups? My terms are (unfortunately) cased.
> > >
> > > What Lookup "sits" behind this suggester
> > > http://search-lucene.com/m/586gA4ccL11
> > > here?
> > >
> > >
> > > > -----Ursprüngliche Nachricht-----
> > > > Von: Dawid Weiss [mailto:dawid.weiss@gmail.com]
> > > > Gesendet: Donnerstag, 5. Mai 2011 15:00
> > > > An: java-user@lucene.apache.org
> > > > Betreff: Re: Using Solr's (Auto)suggest with plain lucene
> > > >
> > > > If you check out the source code of solr/lucene, look at FSTLookup
> > > > class
> > > and
> > > > FSTLookupTest -- you can populate FSTLookup manually with terms/
> > > > phrases from your index and then use the resulting automaton for
> > suggestions.
> > > >
> > > > Dawid
> > > >
> > > > On Thu, May 5, 2011 at 2:54 PM, Clemens Wyss
> > <cl...@mysign.ch>
> > > > wrote:
> > > >
> > > > > I have implemented my index (in fact it's a plugable indexing API)
> > > > > in "plain Lucene". It tried to implement a term suggestion
> > > > > mechanism on my own, being not to happy so far.
> > > > > At
> > > > > http://search-
> > > > lucene.com/m/0QBv41ssGlh/suggestion&subj=Auto+Suggest
> > > > > I have seen Solr's auto suggestion for search terms. Appart from
> > > > > not being "fuzzy" it looks quite promising.
> > > > >
> > > > > How easily can I Integrate Solr's suggestion with my Lucene index?
> > > > >
> > > > > Thx
> > > > > Clemens
> > > > >
> > > > > ------------------------------------------------------------------
> > > > > --- To unsubscribe, e-mail:
> > > > > java-user-unsubscribe@lucene.apache.org
> > > > > For additional commands, e-mail: java-user-help@lucene.apache.org
> > > > >
> > > > >
> > >
>
AW: Using Solr's (Auto)suggest with plain lucene
Posted by Clemens Wyss <cl...@mysign.ch>.
Thx Dawid,
Could you be more precise on 2. ? I'd like to leave the suggested terms untouched, i.e. cased.
Clemens
> -----Ursprüngliche Nachricht-----
> Von: Dawid Weiss [mailto:dawid.weiss@gmail.com]
> Gesendet: Freitag, 6. Mai 2011 11:12
> An: java-user@lucene.apache.org
> Betreff: Re: Using Solr's (Auto)suggest with plain lucene
>
> 1. FSTLookup is in the trunk only, it has not been released.
> 2. Case-insensitive lookups are possible, you'd just need to pass normalized
> tokens from which the dictionary is built and then use the same
> normalization at the time you query for suggestions.
> 3. "http://search-lucene.com/m/586gA4ccL11". I have no idea.
>
> Dawid
>
> On Fri, May 6, 2011 at 11:06 AM, Clemens Wyss <cl...@mysign.ch>
> wrote:
>
> > I have come across TSTLookup.
> > In which jar Do I find FSTLookup? Does any of these lookup-classes
> > allow case insensitive lookups? My terms are (unfortunately) cased.
> >
> > What Lookup "sits" behind this suggester
> > http://search-lucene.com/m/586gA4ccL11
> > here?
> >
> >
> > > -----Ursprüngliche Nachricht-----
> > > Von: Dawid Weiss [mailto:dawid.weiss@gmail.com]
> > > Gesendet: Donnerstag, 5. Mai 2011 15:00
> > > An: java-user@lucene.apache.org
> > > Betreff: Re: Using Solr's (Auto)suggest with plain lucene
> > >
> > > If you check out the source code of solr/lucene, look at FSTLookup
> > > class
> > and
> > > FSTLookupTest -- you can populate FSTLookup manually with terms/
> > > phrases from your index and then use the resulting automaton for
> suggestions.
> > >
> > > Dawid
> > >
> > > On Thu, May 5, 2011 at 2:54 PM, Clemens Wyss
> <cl...@mysign.ch>
> > > wrote:
> > >
> > > > I have implemented my index (in fact it's a plugable indexing API)
> > > > in "plain Lucene". It tried to implement a term suggestion
> > > > mechanism on my own, being not to happy so far.
> > > > At
> > > > http://search-
> > > lucene.com/m/0QBv41ssGlh/suggestion&subj=Auto+Suggest
> > > > I have seen Solr's auto suggestion for search terms. Appart from
> > > > not being "fuzzy" it looks quite promising.
> > > >
> > > > How easily can I Integrate Solr's suggestion with my Lucene index?
> > > >
> > > > Thx
> > > > Clemens
> > > >
> > > > ------------------------------------------------------------------
> > > > --- To unsubscribe, e-mail:
> > > > java-user-unsubscribe@lucene.apache.org
> > > > For additional commands, e-mail: java-user-help@lucene.apache.org
> > > >
> > > >
> >
Re: Using Solr's (Auto)suggest with plain lucene
Posted by Dawid Weiss <da...@gmail.com>.
1. FSTLookup is in the trunk only, it has not been released.
2. Case-insensitive lookups are possible, you'd just need to pass normalized
tokens from which the dictionary is built and then use the same
normalization at the time you query for suggestions.
3. "http://search-lucene.com/m/586gA4ccL11". I have no idea.
Dawid
On Fri, May 6, 2011 at 11:06 AM, Clemens Wyss <cl...@mysign.ch> wrote:
> I have come across TSTLookup.
> In which jar Do I find FSTLookup? Does any of these lookup-classes allow
> case insensitive lookups? My terms are (unfortunately) cased.
>
> What Lookup "sits" behind this suggester
> http://search-lucene.com/m/586gA4ccL11
> here?
>
>
> > -----Ursprüngliche Nachricht-----
> > Von: Dawid Weiss [mailto:dawid.weiss@gmail.com]
> > Gesendet: Donnerstag, 5. Mai 2011 15:00
> > An: java-user@lucene.apache.org
> > Betreff: Re: Using Solr's (Auto)suggest with plain lucene
> >
> > If you check out the source code of solr/lucene, look at FSTLookup class
> and
> > FSTLookupTest -- you can populate FSTLookup manually with terms/ phrases
> > from your index and then use the resulting automaton for suggestions.
> >
> > Dawid
> >
> > On Thu, May 5, 2011 at 2:54 PM, Clemens Wyss <cl...@mysign.ch>
> > wrote:
> >
> > > I have implemented my index (in fact it's a plugable indexing API) in
> > > "plain Lucene". It tried to implement a term suggestion mechanism on
> > > my own, being not to happy so far.
> > > At
> > > http://search-
> > lucene.com/m/0QBv41ssGlh/suggestion&subj=Auto+Suggest
> > > I have seen Solr's auto suggestion for search terms. Appart from not
> > > being "fuzzy" it looks quite promising.
> > >
> > > How easily can I Integrate Solr's suggestion with my Lucene index?
> > >
> > > Thx
> > > Clemens
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > > For additional commands, e-mail: java-user-help@lucene.apache.org
> > >
> > >
>
AW: Using Solr's (Auto)suggest with plain lucene
Posted by Clemens Wyss <cl...@mysign.ch>.
I have come across TSTLookup.
In which jar Do I find FSTLookup? Does any of these lookup-classes allow case insensitive lookups? My terms are (unfortunately) cased.
What Lookup "sits" behind this suggester
http://search-lucene.com/m/586gA4ccL11
here?
> -----Ursprüngliche Nachricht-----
> Von: Dawid Weiss [mailto:dawid.weiss@gmail.com]
> Gesendet: Donnerstag, 5. Mai 2011 15:00
> An: java-user@lucene.apache.org
> Betreff: Re: Using Solr's (Auto)suggest with plain lucene
>
> If you check out the source code of solr/lucene, look at FSTLookup class and
> FSTLookupTest -- you can populate FSTLookup manually with terms/ phrases
> from your index and then use the resulting automaton for suggestions.
>
> Dawid
>
> On Thu, May 5, 2011 at 2:54 PM, Clemens Wyss <cl...@mysign.ch>
> wrote:
>
> > I have implemented my index (in fact it's a plugable indexing API) in
> > "plain Lucene". It tried to implement a term suggestion mechanism on
> > my own, being not to happy so far.
> > At
> > http://search-
> lucene.com/m/0QBv41ssGlh/suggestion&subj=Auto+Suggest
> > I have seen Solr's auto suggestion for search terms. Appart from not
> > being "fuzzy" it looks quite promising.
> >
> > How easily can I Integrate Solr's suggestion with my Lucene index?
> >
> > Thx
> > Clemens
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
Re: Using Solr's (Auto)suggest with plain lucene
Posted by Dawid Weiss <da...@gmail.com>.
If you check out the source code of solr/lucene, look at FSTLookup class and
FSTLookupTest -- you can populate FSTLookup manually with terms/ phrases
from your index and then use the resulting automaton for suggestions.
Dawid
On Thu, May 5, 2011 at 2:54 PM, Clemens Wyss <cl...@mysign.ch> wrote:
> I have implemented my index (in fact it's a plugable indexing API) in
> "plain Lucene". It tried to implement a term suggestion mechanism on my own,
> being not to happy so far.
> At
> http://search-lucene.com/m/0QBv41ssGlh/suggestion&subj=Auto+Suggest
> I have seen Solr's auto suggestion for search terms. Appart from not being
> "fuzzy" it looks quite promising.
>
> How easily can I Integrate Solr's suggestion with my Lucene index?
>
> Thx
> Clemens
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: Using Solr's (Auto)suggest with plain lucene
Posted by Michael McCandless <lu...@mikemccandless.com>.
Also, have a look at the patch on this issue:
https://issues.apache.org/jira/browse/LUCENE-2995
That issue factors out spell checking / auto suggest from Lucene &
Solr into a shared module.
Mike
http://blog.mikemccandless.com
On Thu, May 5, 2011 at 8:54 AM, Clemens Wyss <cl...@mysign.ch> wrote:
> I have implemented my index (in fact it's a plugable indexing API) in "plain Lucene". It tried to implement a term suggestion mechanism on my own, being not to happy so far.
> At
> http://search-lucene.com/m/0QBv41ssGlh/suggestion&subj=Auto+Suggest
> I have seen Solr's auto suggestion for search terms. Appart from not being "fuzzy" it looks quite promising.
>
> How easily can I Integrate Solr's suggestion with my Lucene index?
>
> Thx
> Clemens
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org