You are viewing a plain text version of this content. The canonical link for it is here.

Posted to solr-user@lucene.apache.org by Angel Todorov <at...@gmail.com> on 2017/06/25 21:42:00 UTC

SOLR Suggester returns either the full field value or single terms only

Hi guys,

I am trying to configure the Suggester in a way that i get google-style
auto suggestions:

- I don't want the suggestions to be the _whole_ field value
- I don't want the suggestions to be single terms

For example, if I have a field that has the value "The brown fox jumped
over the fence"

and I type "br" for example, I would like the get things like "Brown" and
"brown fox", but not the whole sentence. Also, i don't want my results to
be just single terms, but to also include phrases.

I have tried a lot of configurations and have found out that if I use the
Document Dictionary Factory, I get the whole field value as a result, If I
use the Fuzzy Dictionary, I only get single terms as results. Nothing
similar to my requirements.  My config is properly configured and my field
type is stored, because i am getting results, it's just that the results
are not what I'd expect.

Would greatly appreciate if you can guide me to the right config.

Thanks,
Angel

Re: SOLR Suggester returns either the full field value or single terms only

Posted by "Mark H. Wood" <mw...@iupui.edu>.

On Wed, Jun 19, 2019 at 12:20:43PM -0700, ppunet wrote:
> As the SuggeterComponent provides the 'entire content' of the field in the
> suggestions. How is it possible to have Suggester to return only part of the
> content of the field, instead of the entire content, which in my scenario
> quite long?

Possibly worthless newbie suggestion:  could you use highlighting to
locate the text that triggered the suggestion, and just chop off
leading and trailing context down to a reasonable length surrounding
the match?  Kind of like you'd see in a printed KWIC index:  give as
much context as will fit the available space, and don't worry about
the rest.

-- 
Mark H. Wood
Lead Technology Analyst

University Library
Indiana University - Purdue University Indianapolis
755 W. Michigan Street
Indianapolis, IN 46202
317-274-0749
www.ulib.iupui.edu

Re: SOLR Suggester returns either the full field value or single terms only

Posted by ppunet <pp...@gmail.com>.

Hey,

As the SuggeterComponent provides the 'entire content' of the field in the
suggestions. How is it possible to have Suggester to return only part of the
content of the field, instead of the entire content, which in my scenario
quite long?



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: SOLR Suggester returns either the full field value or single terms only

Posted by Angel Todorov <at...@gmail.com>.

Hi Alessandro,

Thanks. I've experimented a bit more and here is what I have discovered -
 If my query is enclosed with quotes, then i get multi terms, if it is not
enclosed in quotes, i only get single terms,

Example: will only return single terms:

http://localhost:8080/solr/<core
name>/suggest?suggest=true&suggest.dictionary=mySuggester&wt=json&suggest.q=
*video*

Sample result:

{"responseHeader":{"status":0,"QTime":32},"suggest":{"mySuggester":{"video":{"numFound":10,"suggestions":[{"term":"video","weight":4169589427484439,"payload":""},{"term":"videos","weight":274540867653296,"payload":""},{"term":"videopilot5011pilot2016","weight":137270433826648,"payload":""},{"term":"videoplaylisthttp","weight":137270433826648,"payload":""},{"term":"videotransition","weight":34317608456662,"payload":""}]
.... }}}}

Example: multiterm results

http://localhost:8080/solr/<core
name>/suggest?suggest=true&suggest.dictionary=mySuggester&wt=json&suggest.q=
*"video"*

{"responseHeader":{"status":0,"QTime":12},"suggest":{"mySuggester":{"\"video\"":{"numFound":10,"suggestions":[{"term":"video
shows","weight":3491976244405923328,"payload":""},{"term":"video
from","weight":948906588153783552,"payload":""},{"term":"video
leaving","weight":189781317630756704,"payload":""},{"term":"video
entitled","weight":151825054104605376,"payload":""}}] .... }}}}

What's more , if my query is something like "video g", i get results that
don't include "video", for example:

http://localhost:8080/solr/<core
name>/suggest?suggest=true&suggest.dictionary=mySuggester&wt=json&suggest.q=*"video
g"*

{"responseHeader":{"status":0,"QTime":48},"suggest":{"mySuggester":{"\"video
g\"":{"numFound":10,"suggestions":[{"term":"g
eo","weight":952090016707589760,"payload":""},{"term":"g
em","weight":297528130221121792,"payload":""},{"term":"g
spokesperson","weight":297528130221121792,"payload":""},{"term":"g
prepares","weight":238022504176897440,"payload":""}]}}}}

and so on. I am not sure why it's working in this way, but it doesn't seem
right to me. I am running on SOLR 5.1, if it makes any difference. Could it
be the solr version?

Thanks again,
Angel


On Tue, Jun 27, 2017 at 11:43 AM, alessandro.benedetti <a.benedetti@sease.io
> wrote:

> Hi Angel,
> can you give me an example of query, a couple of documents of example, and
> the suggestions you get ( which you don't expect) ?
>
> The config seems fine ( I remember there were some tricky problems with the
> default separator, but a space should be fine there).
>
> Cheers
>
>
>
> -----
> ---------------
> Alessandro Benedetti
> Search Consultant, R&D Software Engineer, Director
> Sease Ltd. - www.sease.io
> --
> View this message in context: http://lucene.472066.n3.
> nabble.com/SOLR-Suggester-returns-either-the-full-field-
> value-or-single-terms-only-tp4342763p4342987.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>

Re: SOLR Suggester returns either the full field value or single terms only

Posted by "alessandro.benedetti" <a....@sease.io>.

Hi Angel, 
can you give me an example of query, a couple of documents of example, and
the suggestions you get ( which you don't expect) ?

The config seems fine ( I remember there were some tricky problems with the
default separator, but a space should be fine there).

Cheers



-----
---------------
Alessandro Benedetti
Search Consultant, R&D Software Engineer, Director
Sease Ltd. - www.sease.io
--
View this message in context: http://lucene.472066.n3.nabble.com/SOLR-Suggester-returns-either-the-full-field-value-or-single-terms-only-tp4342763p4342987.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: SOLR Suggester returns either the full field value or single terms only

Posted by Angel Todorov <at...@gmail.com>.

hi Alessandro,

Thanks very much. I've read your article - which is really great by the
way! , and configured my suggester in the following way:

  <searchComponent name="suggest" class="solr.SuggestComponent">

  <lst name="suggester">

      <str name="name">mySuggester</str>

      <str name="lookupImpl">FreeTextLookupFactory</str>

      <str name="dictionaryImpl">DocumentDictionaryFactory</str>

      <str name="field">content_suggest</str>

      <str name="buildOnStartup">true</str>

      <str name="ngrams">3</str>

      <str name="separator"> </str>

      <str name="suggestFreeTextAnalyzerFieldType">text_general</str>

    </lst>

  </searchComponent>


The issue is that i am still getting single term suggestions. Sometimes i
am getting multiple terms but they are not separated by a space, but by
other characters like _ and :, for example "breaking_news" or
"breaking:news", whenever i type "brea" or "br".

Do you see anything wrong with my setup ? Thank you

Angel

On Mon, Jun 26, 2017 at 6:04 PM, govind nitk <go...@gmail.com> wrote:

> Hi Alessandro,
>
> Thanks for clarification.
>
>
>
> On Mon, Jun 26, 2017 at 4:53 PM, alessandro.benedetti <
> a.benedetti@sease.io>
> wrote:
>
> > " Don't use an heavy Analyzers, the suggested terms will come from the
> > index,
> > so be sure they are meaningful tokens. A really basic analyser is
> > suggested,
> > stop words and stemming are not "
> >
> > This means that your suggestions will come from the index, so if you use
> > heavy analysers you can get terms suggested which are not really useful :
> >
> > e.g.
> >
> > Solr is an amazing search engine
> >
> > If you have some stemmer in your analysis chain, you will have this
> > behavior
> > :
> >
> > q= ama
> > result : amaz search engin
> >
> > So it is better to have this lookup strategy configured on top of a light
> > analysed field ( or copyfield).
> >
> >
> >
> >
> >
> > -----
> > ---------------
> > Alessandro Benedetti
> > Search Consultant, R&D Software Engineer, Director
> > Sease Ltd. - www.sease.io
> > --
> > View this message in context: http://lucene.472066.n3.
> > nabble.com/SOLR-Suggester-returns-either-the-full-field-
> > value-or-single-terms-only-tp4342763p4342807.html
> > Sent from the Solr - User mailing list archive at Nabble.com.
> >
>

Re: SOLR Suggester returns either the full field value or single terms only

Posted by govind nitk <go...@gmail.com>.

Hi Alessandro,

Thanks for clarification.



On Mon, Jun 26, 2017 at 4:53 PM, alessandro.benedetti <a....@sease.io>
wrote:

> " Don't use an heavy Analyzers, the suggested terms will come from the
> index,
> so be sure they are meaningful tokens. A really basic analyser is
> suggested,
> stop words and stemming are not "
>
> This means that your suggestions will come from the index, so if you use
> heavy analysers you can get terms suggested which are not really useful :
>
> e.g.
>
> Solr is an amazing search engine
>
> If you have some stemmer in your analysis chain, you will have this
> behavior
> :
>
> q= ama
> result : amaz search engin
>
> So it is better to have this lookup strategy configured on top of a light
> analysed field ( or copyfield).
>
>
>
>
>
> -----
> ---------------
> Alessandro Benedetti
> Search Consultant, R&D Software Engineer, Director
> Sease Ltd. - www.sease.io
> --
> View this message in context: http://lucene.472066.n3.
> nabble.com/SOLR-Suggester-returns-either-the-full-field-
> value-or-single-terms-only-tp4342763p4342807.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>

Re: SOLR Suggester returns either the full field value or single terms only

Posted by "alessandro.benedetti" <a....@sease.io>.

" Don't use an heavy Analyzers, the suggested terms will come from the index,
so be sure they are meaningful tokens. A really basic analyser is suggested,
stop words and stemming are not "

This means that your suggestions will come from the index, so if you use
heavy analysers you can get terms suggested which are not really useful :

e.g.

Solr is an amazing search engine

If you have some stemmer in your analysis chain, you will have this behavior
:

q= ama
result : amaz search engin

So it is better to have this lookup strategy configured on top of a light
analysed field ( or copyfield).





-----
---------------
Alessandro Benedetti
Search Consultant, R&D Software Engineer, Director
Sease Ltd. - www.sease.io
--
View this message in context: http://lucene.472066.n3.nabble.com/SOLR-Suggester-returns-either-the-full-field-value-or-single-terms-only-tp4342763p4342807.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: SOLR Suggester returns either the full field value or single terms only

Posted by govind nitk <go...@gmail.com>.

Hi alessandro,
Really nice article.

Can you brief us on "*Don't use an heavy Analyzers*" ?


Regards,
Govind

On Mon, Jun 26, 2017 at 2:19 PM, alessandro.benedetti <a....@sease.io>
wrote:

> Hi Angel,
> your are looking for the Free Text lookup approach.
> You find more info in [1] and [2]
>
> [1]
> https://lucene.apache.org/solr/guide/6_6/suggester.html#Suggester-
> FreeTextLookupFactory
> [2] http://alexbenedetti.blogspot.co.uk/2015/07/solr-you-complete-me.html
>
>
>
> -----
> ---------------
> Alessandro Benedetti
> Search Consultant, R&D Software Engineer, Director
> Sease Ltd. - www.sease.io
> --
> View this message in context: http://lucene.472066.n3.
> nabble.com/SOLR-Suggester-returns-either-the-full-field-
> value-or-single-terms-only-tp4342763p4342790.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>

Re: SOLR Suggester returns either the full field value or single terms only

Posted by "alessandro.benedetti" <a....@sease.io>.

Hi Angel,
your are looking for the Free Text lookup approach.
You find more info in [1] and [2]

[1]
https://lucene.apache.org/solr/guide/6_6/suggester.html#Suggester-FreeTextLookupFactory
[2] http://alexbenedetti.blogspot.co.uk/2015/07/solr-you-complete-me.html



-----
---------------
Alessandro Benedetti
Search Consultant, R&D Software Engineer, Director
Sease Ltd. - www.sease.io
--
View this message in context: http://lucene.472066.n3.nabble.com/SOLR-Suggester-returns-either-the-full-field-value-or-single-terms-only-tp4342763p4342790.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: SOLR Suggester returns either the full field value or single terms only

Posted by govind nitk <go...@gmail.com>.

Hi Angel,

Please Look at these documents.
1. https://home.apache.org/~ctargett/RefGuidePOC/jekyll-full/suggester.html
2.
https://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters#solr.ShingleFilterFactory


Regards,
Govind



On Mon, Jun 26, 2017 at 3:12 AM, Angel Todorov <at...@gmail.com> wrote:

> Hi guys,
>
> I am trying to configure the Suggester in a way that i get google-style
> auto suggestions:
>
> - I don't want the suggestions to be the _whole_ field value
> - I don't want the suggestions to be single terms
>
> For example, if I have a field that has the value "The brown fox jumped
> over the fence"
>
> and I type "br" for example, I would like the get things like "Brown" and
> "brown fox", but not the whole sentence. Also, i don't want my results to
> be just single terms, but to also include phrases.
>
> I have tried a lot of configurations and have found out that if I use the
> Document Dictionary Factory, I get the whole field value as a result, If I
> use the Fuzzy Dictionary, I only get single terms as results. Nothing
> similar to my requirements.  My config is properly configured and my field
> type is stored, because i am getting results, it's just that the results
> are not what I'd expect.
>
> Would greatly appreciate if you can guide me to the right config.
>
> Thanks,
> Angel
>