You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Sandeep Khanzode <sa...@yahoo.com.INVALID> on 2016/09/08 09:35:02 UTC

StrField with Wildcard Search

Hello,
There are quite a few links that detail the difference between StrField and TextField. Also links that explain that, even though the field is indexed, it is not tokenized and stored as a single keyword, as can be verified by the debug analysis on Solr admin and CURL debugQuery options.
What I am unable to understand is how a wildcard works on StrFields? For example, if the name is "John Doe" and I search for "John*", I get that match. Which means, that somewhere deep within, maybe a Trie or Dictionary representation exists that allows this search with a partial string.
I would have assumed that wildcard would match on TextFields which allow (Edge)NGramFilters, etc.  -- SRK

Re: StrField with Wildcard Search

Posted by Ahmet Arslan <io...@yahoo.com.INVALID>.
Hi,


I think AutomatonQuery is used.
http://opensourceconnections.com/blog/2013/02/21/lucene-4-finite-state-automaton-in-10-minutes-intro-tutorial/
https://lucene.apache.org/core/5_4_0/core/org/apache/lucene/search/AutomatonQuery.html

Ahmet


On Thursday, September 8, 2016 3:54 PM, Sandeep Khanzode <sa...@yahoo.com> wrote:



Hi,

Okay.

So it seems that the wildcard searches will perform a (sort-of) dictionary search where they will inspect every (full keyword) token at search time, and do a match instead of a match on pre-created index-time tokens with TextField. However, the wildcard/fuzzy functionality will still be provided no matter the approach...
 
SRK


On Thursday, September 8, 2016 5:05 PM, Ahmet Arslan <io...@yahoo.com.INVALID> wrote:



Hi,

EdgeNGram and Wildcard may be used to achieve the same goal: prefix search or starts with search.

Lets say, wildcard enumerates the whole inverted index, thus it may get slower for very large databases.
With this one no index time manipulation is required.

EdgeNGram does its magic at index time, indexes a lot of tokens, all possible prefixes.
Index size gets bigger, query time no wildcard operator required in this one.

Ahmet




On Thursday, September 8, 2016 12:35 PM, Sandeep Khanzode <sa...@yahoo.com.INVALID> wrote:
Hello,
There are quite a few links that detail the difference between StrField and TextField. Also links that explain that, even though the field is indexed, it is not tokenized and stored as a single keyword, as can be verified by the debug analysis on Solr admin and CURL debugQuery options.
What I am unable to understand is how a wildcard works on StrFields? For example, if the name is "John Doe" and I search for "John*", I get that match. Which means, that somewhere deep within, maybe a Trie or Dictionary representation exists that allows this search with a partial string.
I would have assumed that wildcard would match on TextFields which allow (Edge)NGramFilters, etc.  -- SRK 

Re: StrField with Wildcard Search

Posted by Sandeep Khanzode <sa...@yahoo.com.INVALID>.
Hi,
Okay.
So it seems that the wildcard searches will perform a (sort-of) dictionary search where they will inspect every (full keyword) token at search time, and do a match instead of a match on pre-created index-time tokens with TextField. However, the wildcard/fuzzy functionality will still be provided no matter the approach... SRK 

    On Thursday, September 8, 2016 5:05 PM, Ahmet Arslan <io...@yahoo.com.INVALID> wrote:
 

 Hi,

EdgeNGram and Wildcard may be used to achieve the same goal: prefix search or starts with search.

Lets say, wildcard enumerates the whole inverted index, thus it may get slower for very large databases.
With this one no index time manipulation is required.

EdgeNGram does its magic at index time, indexes a lot of tokens, all possible prefixes.
Index size gets bigger, query time no wildcard operator required in this one.

Ahmet



On Thursday, September 8, 2016 12:35 PM, Sandeep Khanzode <sa...@yahoo.com.INVALID> wrote:
Hello,
There are quite a few links that detail the difference between StrField and TextField. Also links that explain that, even though the field is indexed, it is not tokenized and stored as a single keyword, as can be verified by the debug analysis on Solr admin and CURL debugQuery options.
What I am unable to understand is how a wildcard works on StrFields? For example, if the name is "John Doe" and I search for "John*", I get that match. Which means, that somewhere deep within, maybe a Trie or Dictionary representation exists that allows this search with a partial string.
I would have assumed that wildcard would match on TextFields which allow (Edge)NGramFilters, etc.  -- SRK 


   

Re: StrField with Wildcard Search

Posted by Ahmet Arslan <io...@yahoo.com.INVALID>.
Hi,

EdgeNGram and Wildcard may be used to achieve the same goal: prefix search or starts with search.

Lets say, wildcard enumerates the whole inverted index, thus it may get slower for very large databases.
With this one no index time manipulation is required.

EdgeNGram does its magic at index time, indexes a lot of tokens, all possible prefixes.
Index size gets bigger, query time no wildcard operator required in this one.

Ahmet



On Thursday, September 8, 2016 12:35 PM, Sandeep Khanzode <sa...@yahoo.com.INVALID> wrote:
Hello,
There are quite a few links that detail the difference between StrField and TextField. Also links that explain that, even though the field is indexed, it is not tokenized and stored as a single keyword, as can be verified by the debug analysis on Solr admin and CURL debugQuery options.
What I am unable to understand is how a wildcard works on StrFields? For example, if the name is "John Doe" and I search for "John*", I get that match. Which means, that somewhere deep within, maybe a Trie or Dictionary representation exists that allows this search with a partial string.
I would have assumed that wildcard would match on TextFields which allow (Edge)NGramFilters, etc.  -- SRK