question on tokenization control

[kfdroid <kf...@gmail.com>]

I have a field that is defined using what I believe is fairly standard "text"
fieldType. I have documents with the words 'evaluate', 'evaluating',
'evaluation' in them. When I search on the whole word, obviously it works,
if I search on 'eval' it finds nothing. However for some reason if I search
on 'evalu' it finds all the matches.  Is that an indexing setting or query
setting that will tokenize 'evalu' but not 'eval' and how do I get 'eval' to
be a match? 

Thanks, 
Ken

--
View this message in context: http://lucene.472066.n3.nabble.com/question-on-tokenization-control-tp3953550.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: question on tokenization control

[Walter Underwood <wu...@wunderwood.org>]

Use synonyms at index time. Make "eval" and "evaluate" equivalent words.

wunder

On May 1, 2012, at 1:31 PM, Dan Tuffery wrote:

> Hi,
> 
> "Is that an indexing setting or query setting that will tokenize 'evalu'
> but not 'eval'?"
> 
> Without seeing the tokenizers you're using for the field type it's hard to
> say. You can use Solr's analysis page to see the tokens that are generated
> by the tokenizers in your analysis chain at both query time and index time.
> 
> http://localhost:8983/solr/admin/analysis.jsp
> 
> "how do I get 'eval' to be a match?"
> 
> You could use synonyms to map 'eval' to 'evaluation'.
> 
> Dan
> 
> On Tue, May 1, 2012 at 8:17 PM, kfdroid <kf...@gmail.com> wrote:
> 
>> I have a field that is defined using what I believe is fairly standard
>> "text"
>> fieldType. I have documents with the words 'evaluate', 'evaluating',
>> 'evaluation' in them. When I search on the whole word, obviously it works,
>> if I search on 'eval' it finds nothing. However for some reason if I search
>> on 'evalu' it finds all the matches.  Is that an indexing setting or query
>> setting that will tokenize 'evalu' but not 'eval' and how do I get 'eval'
>> to
>> be a match?
>> 
>> Thanks,
>> Ken
>> 
>> --
>> View this message in context:
>> http://lucene.472066.n3.nabble.com/question-on-tokenization-control-tp3953550.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>> 

--
Walter Underwood
wunder@wunderwood.org

Re: question on tokenization control

[Dan Tuffery <da...@gmail.com>]

Hi,

"Is that an indexing setting or query setting that will tokenize 'evalu'
but not 'eval'?"

Without seeing the tokenizers you're using for the field type it's hard to
say. You can use Solr's analysis page to see the tokens that are generated
by the tokenizers in your analysis chain at both query time and index time.

http://localhost:8983/solr/admin/analysis.jsp

"how do I get 'eval' to be a match?"

You could use synonyms to map 'eval' to 'evaluation'.

Dan

On Tue, May 1, 2012 at 8:17 PM, kfdroid <kf...@gmail.com> wrote:

> I have a field that is defined using what I believe is fairly standard
> "text"
> fieldType. I have documents with the words 'evaluate', 'evaluating',
> 'evaluation' in them. When I search on the whole word, obviously it works,
> if I search on 'eval' it finds nothing. However for some reason if I search
> on 'evalu' it finds all the matches.  Is that an indexing setting or query
> setting that will tokenize 'evalu' but not 'eval' and how do I get 'eval'
> to
> be a match?
>
> Thanks,
> Ken
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/question-on-tokenization-control-tp3953550.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>