You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Gau <ga...@gmail.com> on 2012/05/30 22:39:22 UTC
Difference between textfield and strfield
Hi,
Can anyone explain me the basic pros and cons between textfield and
strfield. I am trying to use Levenstein distance on textfield, but it seems
that it can only be applied on the strfield. So my question is whats the
difference between the 2 and what are the radical advantages of one over the
other
Currently I have the text field defined for first_name and i apply synonym
expansion at query time to this field.
--
View this message in context: http://lucene.472066.n3.nabble.com/Difference-between-textfield-and-strfield-tp3986916.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Difference between textfield and strfield
Posted by Gau <ga...@gmail.com>.
is there any other option to sorting. I mean, sorting can affect query
performance. Is there a way to embed this into Solr and not have a toll on
the system,
I tried boosting the scores based on strdist, but that seems to bring in
more results than expected.
--
View this message in context: http://lucene.472066.n3.nabble.com/Difference-between-textfield-and-strfield-tp3986916p3987338.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Difference between textfield and strfield
Posted by Jack Krupansky <ja...@basetechnology.com>.
I should have said that you need to have both a text and string copy of the
field. Make sure one of them is "stored" and do a copyField to the other
field. Then your main query can be against the text field and the sort
function query against the string field.
-- Jack Krupansky
-----Original Message-----
From: Gau
Sent: Wednesday, May 30, 2012 6:23 PM
To: solr-user@lucene.apache.org
Subject: Re: Difference between textfield and strfield
I cannot move from textfield to strfield, since I am using synonym
expansion.
Is there anything we can do on textfield itself
--
View this message in context:
http://lucene.472066.n3.nabble.com/Difference-between-textfield-and-strfield-tp3986916p3986938.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Difference between textfield and strfield
Posted by Gau <ga...@gmail.com>.
I cannot move from textfield to strfield, since I am using synonym expansion.
Is there anything we can do on textfield itself
--
View this message in context: http://lucene.472066.n3.nabble.com/Difference-between-textfield-and-strfield-tp3986916p3986938.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Difference between textfield and strfield
Posted by Jack Krupansky <ja...@basetechnology.com>.
Take a look at "strdist" and see whether that does enough of what you want:
See:
http://wiki.apache.org/solr/FunctionQuery#strdist
Some discussion here:
http://lucene.472066.n3.nabble.com/can-t-use-strdist-as-functionquery-td1023390.html
And these would need to be "string" fields, not "text" since it is using the
full, literal character string value.
-- Jack Krupansky
-----Original Message-----
From: Gau
Sent: Wednesday, May 30, 2012 5:23 PM
To: solr-user@lucene.apache.org
Subject: Re: Difference between textfield and strfield
Well the I do not have phrases for synonym expansion. So it does work well.
The synonym expansion is done at query time. And since i am just searching
against the first name field, tf, idf and other ranking parameters do not
make sense, hence their weight has been initialized to 1. So after applying
synonym expansion I am getting results in random word format.
the Results are perfect just that they are not ordered by Levenstein
distance of the original query.
So the use case is
if use enters query ab
it gets expanded at query time to abc,abxy,aberfg
And I get results for ab, abc, abxy, aberfg.
But I want them to be sorted by Levenstein distance from the original query
(ab)
So order shoud be
ab
abc
abxy
aberfg
...... !
TextField makes this even more difficult? Any other suggestions?
Spellcheckers? Ngrams?
--
View this message in context:
http://lucene.472066.n3.nabble.com/Difference-between-textfield-and-strfield-tp3986916p3986928.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Difference between textfield and strfield
Posted by Gau <ga...@gmail.com>.
Well the I do not have phrases for synonym expansion. So it does work well.
The synonym expansion is done at query time. And since i am just searching
against the first name field, tf, idf and other ranking parameters do not
make sense, hence their weight has been initialized to 1. So after applying
synonym expansion I am getting results in random word format.
the Results are perfect just that they are not ordered by Levenstein
distance of the original query.
So the use case is
if use enters query ab
it gets expanded at query time to abc,abxy,aberfg
And I get results for ab, abc, abxy, aberfg.
But I want them to be sorted by Levenstein distance from the original query
(ab)
So order shoud be
ab
abc
abxy
aberfg
...... !
TextField makes this even more difficult? Any other suggestions?
Spellcheckers? Ngrams?
--
View this message in context: http://lucene.472066.n3.nabble.com/Difference-between-textfield-and-strfield-tp3986916p3986928.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Difference between textfield and strfield
Posted by Jack Krupansky <ja...@basetechnology.com>.
A text field is a sequence of terms that has been tokenized while a string
field is a single term (although it can also be multivalued.)
Punctuation and spacing is ignored for text fields. Text tends to be
lowercased, stemmed, and even stop words removed. You tend to search text
using a handful of keywords whose exact order is not required, although
quoted phrases can be used as well. Fuzzy queries can be done on individual
terms (words). Wildcards as well.
String fields are literal character strings with all punctuation, spacing,
and case preserved. Anything other than exact match is done using wildcards,
although I suppose fuzzy query should work as well.
String fields are useful for facets and filter queries or display.
Text fields are useful for keyword search.
Synonyms are a token filtering, which applies to text fields, not string
fields.
A fuzzy query would not work properly for a synonym expansion in which some
of the terms are phrases, but should otherwise work for a text field term.
If you have a specific case that fails, please provide the details.
-- Jack Krupansky
-----Original Message-----
From: Gau
Sent: Wednesday, May 30, 2012 4:39 PM
To: solr-user@lucene.apache.org
Subject: Difference between textfield and strfield
Hi,
Can anyone explain me the basic pros and cons between textfield and
strfield. I am trying to use Levenstein distance on textfield, but it seems
that it can only be applied on the strfield. So my question is whats the
difference between the 2 and what are the radical advantages of one over the
other
Currently I have the text field defined for first_name and i apply synonym
expansion at query time to this field.
--
View this message in context:
http://lucene.472066.n3.nabble.com/Difference-between-textfield-and-strfield-tp3986916.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Difference between textfield and strfield
Posted by manju16832003 <ma...@gmail.com>.
Hey Iori,
Apologize for misunderstanding :-).
Yes agree with you, faceting will be OK with TextField type however I'm
concern about performance impact while running the facets if we have
millions of documents.
I wish in future we could apply tokensizers and filters to String fields.
:-).Thanks for your inputs.
--
View this message in context: http://lucene.472066.n3.nabble.com/Difference-between-textfield-and-strfield-tp3986916p4105841.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Difference between textfield and strfield
Posted by Ahmet Arslan <io...@yahoo.com>.
Hi Manju,
Ahmet is me :) Faceting will be OK with lowercase field type. Even if it is solr.TextField. KeywordTokenizer keeps its input as a single token. Similar behavior as string field. With solr.TextField + KeywordTokenizer you can add further token filters. For example lowercase filter. With string type you cannot add any token filters.
As Erick suggested, you can play with field types at Admin analysis page. It allows you to enter sample text and displays generated tokens visually.
On Sunday, December 8, 2013 2:00 PM, manju16832003 <ma...@gmail.com> wrote:
I don't understand. Use the field type *Ahmet* recommended. Who is Ahmet?
--
View this message in context: http://lucene.472066.n3.nabble.com/Difference-between-textfield-and-strfield-tp3986916p4105570.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Difference between textfield and strfield
Posted by manju16832003 <ma...@gmail.com>.
I don't understand. Use the field type *Ahmet* recommended. Who is Ahmet?
--
View this message in context: http://lucene.472066.n3.nabble.com/Difference-between-textfield-and-strfield-tp3986916p4105570.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Difference between textfield and strfield
Posted by Erick Erickson <er...@gmail.com>.
Use the field type Ahmet recommended. It'll look almost exactly like String
with the exception it'll be case insensitive. Take a look at the
admin/analysis page with this field type and you'll understand why we're
recommending this.
Best,
Erick
On Thu, Dec 5, 2013 at 8:33 PM, manju16832003 <ma...@gmail.com>wrote:
> Hi Iori,
>
> Thank you replying, really appreciate that.
>
> My concern not to use *TextField* and I want to make use of *string* field.
>
> Reason is that I have 7 fields that I want to apply case-insensitiveness
> and
> all these fields are *facetable* fields. It would not be feasible if I
> change data type from String to TextField just to apply
> case-insensitiveness. However Solr document suggests that if we were to
> make
> fields facetable, its better to use String.
>
> Is there any way of achieving this using String itself?.
>
> Thank you,
> Manju
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Difference-between-textfield-and-strfield-tp3986916p4105238.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
Re: Difference between textfield and strfield
Posted by manju16832003 <ma...@gmail.com>.
Hi Iori,
Thank you replying, really appreciate that.
My concern not to use *TextField* and I want to make use of *string* field.
Reason is that I have 7 fields that I want to apply case-insensitiveness and
all these fields are *facetable* fields. It would not be feasible if I
change data type from String to TextField just to apply
case-insensitiveness. However Solr document suggests that if we were to make
fields facetable, its better to use String.
Is there any way of achieving this using String itself?.
Thank you,
Manju
--
View this message in context: http://lucene.472066.n3.nabble.com/Difference-between-textfield-and-strfield-tp3986916p4105238.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Difference between textfield and strfield
Posted by Ahmet Arslan <io...@yahoo.com>.
Hi Manju,
You can use the following type ( taken from example schema.xml) . You may add TrimFilterFactory too.
<!-- lowercases the entire field value, keeping it as a single token. -->
<fieldType name="lowercase" class="solr.TextField" positionIncrementGap="100">
<analyzer>
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory" />
</analyzer>
</fieldType>
On Thursday, December 5, 2013 12:09 PM, manju16832003 <ma...@gmail.com> wrote:
Hi,
If we can not analyse string field, is there any other way to apply case
insensitiveness.
Ex: I have a field called *make* its values are Toyota, Honda, Chery etc. In
Solr make data type is string and values are stored as they appear (Toyota,
Honda, Chery).
However, when user try to search for toyota (instead Toyota), Solr wont
fetch any result set as it does not match the exact word Toyota.
I thought of applying LowerCaseFilterFactory at query level, but as its
string field, I won't be able to do that.
Is there any way, as I would like to search the user with case insensitive.?
--
View this message in context: http://lucene.472066.n3.nabble.com/Difference-between-textfield-and-strfield-tp3986916p4105095.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Difference between textfield and strfield
Posted by manju16832003 <ma...@gmail.com>.
Hi,
If we can not analyse string field, is there any other way to apply case
insensitiveness.
Ex: I have a field called *make* its values are Toyota, Honda, Chery etc. In
Solr make data type is string and values are stored as they appear (Toyota,
Honda, Chery).
However, when user try to search for toyota (instead Toyota), Solr wont
fetch any result set as it does not match the exact word Toyota.
I thought of applying LowerCaseFilterFactory at query level, but as its
string field, I won't be able to do that.
Is there any way, as I would like to search the user with case insensitive.?
--
View this message in context: http://lucene.472066.n3.nabble.com/Difference-between-textfield-and-strfield-tp3986916p4105095.html
Sent from the Solr - User mailing list archive at Nabble.com.