Best practice: Autosuggest/autocomplete vs. "real search"

[Thomas Michael Engelke <th...@posteo.de>]

 

 We're using Solr as a backend for an ECommerce site/system. The Solr
index stores products with selected attributes, as well as a dedicated
field for autocomplete suggestions (Done via AJAX request when typing in
the search box without pressing return).

The autosuggest field is supplied by copyField directives from certain
select product attribute fields (description and/or name mostly). It
uses EdgeNGramFilterFactory to complete words not yet typed completely,
and it works quite well.

However, we come across an issue with a disconnect between the
autosuggest results and results of a "normal search", that is, a query
over the full fields of the product. Let's say there are products that
are called "motor".

- When autosuggesting, typing "mot" autosuggests all products with
"motor", because the EdgeNGram created "m", "mo", "mot", "moto" and
"motor", respectively, and it matches.
- When searching for "mot", however (i.e. pressing enter when seeing the
autosuggestions), it doesn't find any products. The autosuggest field is
not part of the "real" search, and no product attribute contains "mot"
as a word.

One obvious solution would be to incorporate the "autosuggest" field
into the "real" search, however, this adds many tokens to the index that
aren't really part of the products indexed and makes for strange search
results, for example when an NGram is also a word, but the record itself
does contain the search term only as part of a word.

Are there clever solutions to this problem? 

Re: Best practice: Autosuggest/autocomplete vs. "real search"

[Thomas Michael Engelke <th...@posteo.de>]

 The dedicated "autosuggest" field is not used by a suggester component,
instead we just directly query it ("/select"). I'm trying to read my way
into how the suggesters work, and toying around with some configurations
(For instance from here:
http://www.andornot.com/blog/post/Advanced-autocomplete-with-Solr-Ngrams-and-Twitters-typeaheadjs.aspx).

Compared to how you can analyze search result through the Solr backend,
the analysis of suggester results seems to be sorely lacking.

Am 10.11.2014 14:37 schrieb Michael Sokolov: 

> The goal is to ensure that suggestions from autocomplete are actually terms in the main index, so that the suggestions will actually result in matches. You've considered expanding the main index by adding the suggestion n-grams to it, but it would probably be better to alter your suggester so that it produces only tokens that are in the main index. I think this is basically how all the Suggester implementations are designed to work already; are you using one of those, or are you using the TermsComponent, or something else?
> 
> -Mike
> 
> On 11/10/14 2:54 AM, Thomas Michael Engelke wrote:
> 
>> We're using Solr as a backend for an ECommerce site/system. The Solr index stores products with selected attributes, as well as a dedicated field for autocomplete suggestions (Done via AJAX request when typing in the search box without pressing return). The autosuggest field is supplied by copyField directives from certain select product attribute fields (description and/or name mostly). It uses EdgeNGramFilterFactory to complete words not yet typed completely, and it works quite well. However, we come across an issue with a disconnect between the autosuggest results and results of a "normal search", that is, a query over the full fields of the product. Let's say there are products that are called "motor". - When autosuggesting, typing "mot" autosuggests all products with "motor", because the EdgeNGram created "m", "mo", "mot", "moto" and "motor", respectively, and it matches. - When searching for "mot", however (i.e. pressing enter when seeing the autosuggestions), it doesn't
find any products. The autosuggest field is not part of the "real" search, and no product attribute contains "mot" as a word. One obvious solution would be to incorporate the "autosuggest" field into the "real" search, however, this adds many tokens to the index that aren't really part of the products indexed and makes for strange search results, for example when an NGram is also a word, but the record itself does contain the search term only as part of a word. Are there clever solutions to this problem?
 

Re: Best practice: Autosuggest/autocomplete vs. "real search"

[Jorge Luis Betancourt Gonzalez <jl...@uci.cu>]

It wouldn’t be easy if in the site you’ll ensure that only terms are submitted to the actual search? In app I worked some time ago the default behavior of the Javascript component used for autocompletion was to first autocomplete the term in the input and then submit the query against the backend. I know this is not what you’ve asked for but could work? I’m just firing a bullet in the air here! :-)

On Nov 10, 2014, at 8:37 AM, Michael Sokolov <ms...@safaribooksonline.com> wrote:

> The goal is to ensure that suggestions from autocomplete are actually terms in the main index, so that the suggestions will actually result in matches.  You've considered expanding the main index by adding the suggestion n-grams to it, but it would probably be better to alter your suggester so that it produces only tokens that are in the main index.  I think this is basically how all the Suggester implementations are designed to work already; are you using one of those, or are you using the TermsComponent, or something else?
> 
> -Mike
> 
> On 11/10/14 2:54 AM, Thomas Michael Engelke wrote:
>>  
>>  We're using Solr as a backend for an ECommerce site/system. The Solr
>> index stores products with selected attributes, as well as a dedicated
>> field for autocomplete suggestions (Done via AJAX request when typing in
>> the search box without pressing return).
>> 
>> The autosuggest field is supplied by copyField directives from certain
>> select product attribute fields (description and/or name mostly). It
>> uses EdgeNGramFilterFactory to complete words not yet typed completely,
>> and it works quite well.
>> 
>> However, we come across an issue with a disconnect between the
>> autosuggest results and results of a "normal search", that is, a query
>> over the full fields of the product. Let's say there are products that
>> are called "motor".
>> 
>> - When autosuggesting, typing "mot" autosuggests all products with
>> "motor", because the EdgeNGram created "m", "mo", "mot", "moto" and
>> "motor", respectively, and it matches.
>> - When searching for "mot", however (i.e. pressing enter when seeing the
>> autosuggestions), it doesn't find any products. The autosuggest field is
>> not part of the "real" search, and no product attribute contains "mot"
>> as a word.
>> 
>> One obvious solution would be to incorporate the "autosuggest" field
>> into the "real" search, however, this adds many tokens to the index that
>> aren't really part of the products indexed and makes for strange search
>> results, for example when an NGram is also a word, but the record itself
>> does contain the search term only as part of a word.
>> 
>> Are there clever solutions to this problem?
> 

Re: Best practice: Autosuggest/autocomplete vs. "real search"

[Michael Sokolov <ms...@safaribooksonline.com>]

The goal is to ensure that suggestions from autocomplete are actually 
terms in the main index, so that the suggestions will actually result in 
matches.  You've considered expanding the main index by adding the 
suggestion n-grams to it, but it would probably be better to alter your 
suggester so that it produces only tokens that are in the main index.  I 
think this is basically how all the Suggester implementations are 
designed to work already; are you using one of those, or are you using 
the TermsComponent, or something else?

-Mike

On 11/10/14 2:54 AM, Thomas Michael Engelke wrote:
>   
>
>   We're using Solr as a backend for an ECommerce site/system. The Solr
> index stores products with selected attributes, as well as a dedicated
> field for autocomplete suggestions (Done via AJAX request when typing in
> the search box without pressing return).
>
> The autosuggest field is supplied by copyField directives from certain
> select product attribute fields (description and/or name mostly). It
> uses EdgeNGramFilterFactory to complete words not yet typed completely,
> and it works quite well.
>
> However, we come across an issue with a disconnect between the
> autosuggest results and results of a "normal search", that is, a query
> over the full fields of the product. Let's say there are products that
> are called "motor".
>
> - When autosuggesting, typing "mot" autosuggests all products with
> "motor", because the EdgeNGram created "m", "mo", "mot", "moto" and
> "motor", respectively, and it matches.
> - When searching for "mot", however (i.e. pressing enter when seeing the
> autosuggestions), it doesn't find any products. The autosuggest field is
> not part of the "real" search, and no product attribute contains "mot"
> as a word.
>
> One obvious solution would be to incorporate the "autosuggest" field
> into the "real" search, however, this adds many tokens to the index that
> aren't really part of the products indexed and makes for strange search
> results, for example when an NGram is also a word, but the record itself
> does contain the search term only as part of a word.
>
> Are there clever solutions to this problem?