You are viewing a plain text version of this content. The canonical link for it is here.

Posted to solr-user@lucene.apache.org by Norberto Meijome <fr...@meijome.net> on 2008/06/23 08:23:55 UTC

n-Gram, only works with queries of 2 letters

hi there,
my use case : I  want to be able to match documents when only a partial word is provided. ie, searching for 'roc' or 'ock' should match documents containing 'rock'.

As I understand, the way to solve this problem is to use the nGram tokenizer @ index time and the nGram analyser  @ search time. I want at least 'sub terms' of 2 to 6 characters indexed, so I can match substrings that length.

What I have (using 1.3 , nightly build from 2008-06-15):

schema.xml :
[..]
	    <!-- n-gram tokenization -->
    <fieldType name="ngram" class="solr.TextField" positionIncrementGap="100">
      <analyzer type="index">
        <tokenizer class="org.apache.solr.analysis.NGramTokenizerFactory" minGramSize="2" maxGramSize="6"/>
      </analyzer>
      <analyzer type="query">
        <tokenizer class="org.apache.solr.analysis.NGramTokenizerFactory" minGramSize="2" maxGramSize="6"/>
      </analyzer>
    </fieldType>
[..]
		<field name="artist" type="text" indexed="true" stored="true"
			required="true" />

			
		<!--  make this field not stored after testing. -->
		<field name="artist_ngram" type="ngram" indexed="true" stored="true" 
			required="true" />

[...]

	<copyField source="artist" dest="artist_ngram" />

----

SolrConfig is same as from sample app, minus a few comments.

After each change in schema or solrconfig, I delete the index and load the documents (240 of them) again.

when I search :
http://localhost:8983/solr/_test_/select?q=ock&df=artist_ngram&debugQuery=true

I get 0 results:
=====================
−
	<response>
−
	<lst name="responseHeader">
<int name="status">0</int>
<int name="QTime">38</int>
−
	<lst name="params">
<str name="q">ock</str>
<str name="df">artist_ngram</str>
<str name="debugQuery">true</str>
</lst>
</lst>
<result name="response" numFound="0" start="0"/>
−
	<lst name="debug">
<str name="rawquerystring">ock</str>
<str name="querystring">ock</str>
<str name="parsedquery">PhraseQuery(artist_ngram:"oc ck ock")</str>
<str name="parsedquery_toString">artist_ngram:"oc ck ock"</str>
<lst name="explain"/>
−
	<lst name="timing">
<double name="time">37.0</double>
−
	<lst name="prepare">
<double name="time">1.0</double>
−
	<lst name="org.apache.solr.handler.component.QueryComponent">
<double name="time">1.0</double>
</lst>
−
	<lst name="org.apache.solr.handler.component.FacetComponent">
<double name="time">0.0</double>
</lst>
−
	<lst name="org.apache.solr.handler.component.MoreLikeThisComponent">
<double name="time">0.0</double>
</lst>
−
	<lst name="org.apache.solr.handler.component.HighlightComponent">
<double name="time">0.0</double>
</lst>
−
	<lst name="org.apache.solr.handler.component.DebugComponent">
<double name="time">0.0</double>
</lst>
</lst>
−
	<lst name="process">
<double name="time">35.0</double>
−
	<lst name="org.apache.solr.handler.component.QueryComponent">
<double name="time">0.0</double>
</lst>
−
	<lst name="org.apache.solr.handler.component.FacetComponent">
<double name="time">0.0</double>
</lst>
−
	<lst name="org.apache.solr.handler.component.MoreLikeThisComponent">
<double name="time">0.0</double>
</lst>
−
	<lst name="org.apache.solr.handler.component.HighlightComponent">
<double name="time">0.0</double>
</lst>
−
	<lst name="org.apache.solr.handler.component.DebugComponent">
<double name="time">35.0</double>
</lst>
</lst>
</lst>
</lst>
</response>
=====================

The same when I search for 'roc'. BUT if i search for any of the 2-character tokens (ro, oc, ck ), I find everything fine.

===================================
.<response>
−
	<lst name="responseHeader">
<int name="status">0</int>
<int name="QTime">4</int>
−
	<lst name="params">
<str name="q">oc</str>
<str name="df">artist_ngram</str>
<str name="debugQuery">true</str>
</lst>
</lst>
−
	<result name="response" numFound="5" start="0">
−
	<doc>
<str name="artist">Jay Rock</str>
<str name="artist_ngram">Jay Rock</str>
<str name="artistid">Jay Rock</str>
<date name="index_timestamp">2008-06-23T04:46:05.933Z</date>
</doc>
−
[...]
</result>
−
	<lst name="debug">
<str name="rawquerystring">oc</str>
<str name="querystring">oc</str>
<str name="parsedquery">artist_ngram:oc</str>
<str name="parsedquery_toString">artist_ngram:oc</str>
−
	<lst name="explain">
−
	<str name="Jay Rock">

0.87916493 = (MATCH) fieldWeight(artist_ngram:oc in 82), product of:
  1.0 = tf(termFreq(artist_ngram:oc)=1)
  4.6888795 = idf(docFreq=5, numDocs=240)
  0.1875 = fieldNorm(field=artist_ngram, doc=82)
</str>
−
[...]
</response>
===================================

On analysis.jsp I can see that all the tokens are generated, both in index and search:
Fieldname = artist_ngram
field value (Index) = rock
Field Value (query) = ock

(marked with * * are the terms shown as a match)

Index Analyzer 
org.apache.solr.analysis.NGramTokenizerFactory {maxGramSize=6, minGramSize=2}
term position 	1	2	3	4	5	6
term text 	ro	*oc*	*ck*	roc	*ock*	rock
term type 	word	word	word	word	word	word
source start,end 	0,2	1,3	2,4	0,3	1,4	0,4
Query Analyzer
org.apache.solr.analysis.NGramTokenizerFactory {maxGramSize=6, minGramSize=2}
term position 	1	2	3
term text 	oc	ck	ock
term type 	word	word	word
source start,end 	0,2	1,3	0,3

What am I doing wrong here? Why does only the searches for 'oc' , 'ro', 'ck' return matching documents, but 3 or 4 letter terms (rock, even) don't match? 

thanks for any help you can provide,
B
_________________________
{Beto|Norberto|Numard} Meijome

"That's what I love about GUIs: They make simple tasks easier, and complex tasks impossible."
   John William Chambless

I speak for myself, not my employer. Contents may be hot. Slippery when wet. Reading disclaimers makes you go blind. Writing them is worse. You have been Warned.

RE: Can I specify the default operator at query time ?

Posted by Pierre-Yves LANDRON <pl...@hotmail.com>.

Thanks !!!

it'll save me great time !



> From: erik@ehatchersolutions.com
> To: solr-user@lucene.apache.org
> Subject: Re: Can I specify the default operator at query time ?
> Date: Mon, 23 Jun 2008 05:54:13 -0400
> 
> 
> On Jun 23, 2008, at 5:42 AM, Pierre-Yves LANDRON wrote:
> > Is there a way to transgress the default operator defined in  
> > solrConfig for a particular query, and for this query only (a query- 
> > specific option) ?
> 
> Yes, you can specify the q.op=AND or q.op=OR per Solr request.
> 
> 	Erik
> 

_________________________________________________________________
Explore the seven wonders of the world
http://search.msn.com/results.aspx?q=7+wonders+world&mkt=en-US&form=QBRE

Re: Can I specify the default operator at query time ?

Posted by Erik Hatcher <er...@ehatchersolutions.com>.

On Jun 23, 2008, at 5:42 AM, Pierre-Yves LANDRON wrote:
> Is there a way to transgress the default operator defined in  
> solrConfig for a particular query, and for this query only (a query- 
> specific option) ?

Yes, you can specify the q.op=AND or q.op=OR per Solr request.

	Erik

Re: Can I specify the default operator at query time ?

Posted by Chris Hostetter <ho...@fucit.org>.

: Subject: Can I specify the default operator at query time ?
: In-Reply-To: <20...@ayiin>

http://people.apache.org/~hossman/#threadhijack
Thread Hijacking on Mailing Lists

When starting a new discussion on a mailing list, please do not reply to 
an existing message, instead start a fresh email.  Even if you change the 
subject line of your email, other mail headers still track which thread 
you replied to and your question is "hidden" in that thread and gets less 
attention.   It makes following discussions in the mailing list archives 
particularly difficult.
See Also:  http://en.wikipedia.org/wiki/Thread_hijacking




-Hoss

Can I specify the default operator at query time ?

Posted by Pierre-Yves LANDRON <pl...@hotmail.com>.

Hello,

Is there a way to transgress the default operator defined in solrConfig for a particular query, and for this query only (a query-specific option) ?

Thanks,
P-Y Landron

_________________________________________________________________
Discover the new Windows Vista
http://search.msn.com/results.aspx?q=windows+vista&mkt=en-US&form=QBRE

Re: n-Gram, only works with queries of 2 letters

Posted by Norberto Meijome <fr...@meijome.net>.

On Mon, 23 Jun 2008 16:23:55 +1000
Norberto Meijome <fr...@meijome.net> wrote:

> hi there,
> my use case : I  want to be able to match documents when only a partial word is provided. ie, searching for 'roc' or 'ock' should match documents containing 'rock'.
> 
> As I understand, the way to solve this problem is to use the nGram tokenizer @ index time and the nGram analyser  @ search time. I want at least 'sub terms' of 2 to 6 characters indexed, so I can match substrings that length.
> 

Correction - i only get docs returned when the length of my query matches minGramSize. I changed minGramSize to 3 (+ restart service + reindexed) , and 2-letter queries fail (as expected). 3 letter queries start working, but anything larger doesn't. 

B
_________________________
{Beto|Norberto|Numard} Meijome

I used to hate weddings; all the Grandmas would poke me and
say, "You're next sonny!" They stopped doing that when i
started to do it to them at funerals.

I speak for myself, not my employer. Contents may be hot. Slippery when wet. Reading disclaimers makes you go blind. Writing them is worse. You have been Warned.