You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by "Parul Gupta(Knimbus)" <pa...@gmail.com> on 2013/07/11 08:55:55 UTC
Problem using Term Component in solr
Hi All....
I am using *Term component* in solr for searching titles with short form
using wild card characters(.*) and [a-z0-9]*.
I am using *Term Component* specifically as wild card characters are not
working on *select?q=* query search.
Examples of some *title *are:
1)Medicine, Health Care and Philosophy
2)Medical Physics
3)Physics of fluids
4)Medical Engineering and Physics
***When i do *solr query*:
localhost:8080/solr3.6/OA/terms?terms.fl=title&terms.regex=phy.*
fluids&terms.regex.flag=case_insensitive&terms.limit=10
*Output* is 3rd title:
*Physics of fluids*
This is relevant output.
***But when i do *solr query*:
localhost:8080/solr3.6/OA/terms?terms.fl=title&terms.regex=med.*
phy.*&terms.regex.flag=case_insensitive&terms.limit=10
*Output* are 2nd and 4th title:
*Medical Engineering and Physics*
*Medical Physics*
This is irrelevant.I want only one result for this query i.e. *Medical
Physics*
*********************************Although i have changed my wild card
characters to *[a-z0-9]** instead of *.** ,but than first query doesn't work
as '*of*' is included in '*Physics of fluids*'.However Second query works
fine .
example of query is:
localhost:8080/solr3.6/OA/terms?terms.fl=title&terms.regex=med[a-z0-9]*
phy[a-z0-9]*&terms.regex.flag=case_insensitive&terms.limit=10
This works fine,gives one output *Medical Physics*.
If there is another way for searching using *Term Component* or without
using it..Please suggest to neglect such stop words.
Note:Term Component works only on string dataType field. :(
--
View this message in context: http://lucene.472066.n3.nabble.com/Problem-using-Term-Component-in-solr-tp4077200.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Problem using Term Component in solr
Posted by Erick Erickson <er...@gmail.com>.
by "regularizing the title" I meant either indexing and
searching exactly:
Medical Engineering and Physics
or
Medical Eng. and Phys.
Or you could remove the stopwords yourself at both
index and query time, which would fix your "Physics
of Fluids" example.
The problem here is that you'll be forever fiddling with
this and getting it _almost_ right, then the next
anomaly will happen Siiigh
You might actually be much better off with an ngram
or edgeNgram approach. You'd probably want to
tokenize the titles, and perhaps auto-generate phrase
queries...
Best
Erick
On Sun, Jul 14, 2013 at 7:30 AM, Parul Gupta(Knimbus)
<pa...@gmail.com> wrote:
> Hi,
>
> Vocabulary is not known that's the main issue else I will implement synonyms
> instead.
> what do u mean by 'regularizing the title'.
>
> so let me know some solution...
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Problem-using-Term-Component-in-solr-tp4077200p4077865.html
> Sent from the Solr - User mailing list archive at Nabble.com.
Re: Problem using Term Component in solr
Posted by "Parul Gupta(Knimbus)" <pa...@gmail.com>.
Hi,
Vocabulary is not known that's the main issue else I will implement synonyms
instead.
what do u mean by 'regularizing the title'.
so let me know some solution...
--
View this message in context: http://lucene.472066.n3.nabble.com/Problem-using-Term-Component-in-solr-tp4077200p4077865.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Problem using Term Component in solr
Posted by Erick Erickson <er...@gmail.com>.
Is the vocabulary known? That is, do you know the abbreviations that
will be used? If so, you could consider synonyms, in which case you'd
go to tokenized titles and use phrase queries to get your matches...
Regexes often don't scale extremely well, although the 4.x FST
implementations are much faster than they used to be.
It seems to me that regularizing the titles is a better idea than
trying to fake it with regexes, but you know your problem space better
than me...
Best
Erick
On Fri, Jul 12, 2013 at 1:32 PM, Parul Gupta(Knimbus)
<pa...@gmail.com> wrote:
> Hi,
> Ok I will not use Bold text in my queries....
>
> I guess my question is not clear to you....
>
> See what I am doing is, i have a live source say 'A' and a stored database
> say it as 'B'.ok
> A and B ,both have title fields in them.Consider A as non-persistent solr
> and B as persistent solr.
>
> I have to match the title coming from A to the database B.
>
> Since some title from live source A comes in short form e.g 'med. phys.' and
> 'phys. fluids'.
> But corresponding to these titles my database B have titles 'medical
> physics' and 'physics of fluids'.
> Since this type of differences occurs and A not able to search there
> corresponding titles in B by using 'tokenized' field 'title' with using wild
> cards,hence i used Term component first.Which gives me the corresponding
> matched title with B.When i got the full title like 'medical physics',i
> fetched it from HTML,and then again search it in tokenized field of 'title'
> say it 'titlenew'(copy field of title) which brings me result 'medical
> physics'.But I am failing to get match of 'phys. fluids' with 'physics of
> fluids' as it has stop word in it using [a-z0-9]*.
>
> Hope know u will get my issue...and will help..
> thanks..
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Problem-using-Term-Component-in-solr-tp4077200p4077628.html
> Sent from the Solr - User mailing list archive at Nabble.com.
Re: Problem using Term Component in solr
Posted by "Parul Gupta(Knimbus)" <pa...@gmail.com>.
Hi,
Ok I will not use Bold text in my queries....
I guess my question is not clear to you....
See what I am doing is, i have a live source say 'A' and a stored database
say it as 'B'.ok
A and B ,both have title fields in them.Consider A as non-persistent solr
and B as persistent solr.
I have to match the title coming from A to the database B.
Since some title from live source A comes in short form e.g 'med. phys.' and
'phys. fluids'.
But corresponding to these titles my database B have titles 'medical
physics' and 'physics of fluids'.
Since this type of differences occurs and A not able to search there
corresponding titles in B by using 'tokenized' field 'title' with using wild
cards,hence i used Term component first.Which gives me the corresponding
matched title with B.When i got the full title like 'medical physics',i
fetched it from HTML,and then again search it in tokenized field of 'title'
say it 'titlenew'(copy field of title) which brings me result 'medical
physics'.But I am failing to get match of 'phys. fluids' with 'physics of
fluids' as it has stop word in it using [a-z0-9]*.
Hope know u will get my issue...and will help..
thanks..
--
View this message in context: http://lucene.472066.n3.nabble.com/Problem-using-Term-Component-in-solr-tp4077200p4077628.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Problem using Term Component in solr
Posted by Erick Erickson <er...@gmail.com>.
bq: Note:Term Component works only on string dataType field. :(
Not true. Term Component will work on any indexed field. It'll bring
back the _tokens_ that have been indexed though, which are often
individual words so your examples "medical physics" would be two
separate tokens so it may be puzzling.
A general request, please don't put bold text. I know it's an attempt
to help direct attention to the important bits, but (at least in gmail in
my browser) bolds are replaced by "*" before and after, which
especially when looking at wildcard questions is really confusing <G>.
But I have to ask you to back up a bit. _Why_ are you using
TermsComponent to search titles? Why not use Solr for what
it's good for and just search a _tokenized_ title field? This feels
like an XY problem.
Best
Erick
On Thu, Jul 11, 2013 at 2:55 AM, Parul Gupta(Knimbus)
<pa...@gmail.com> wrote:
> Hi All....
>
> I am using *Term component* in solr for searching titles with short form
> using wild card characters(.*) and [a-z0-9]*.
>
> I am using *Term Component* specifically as wild card characters are not
> working on *select?q=* query search.
>
> Examples of some *title *are:
>
> 1)Medicine, Health Care and Philosophy
> 2)Medical Physics
> 3)Physics of fluids
> 4)Medical Engineering and Physics
>
> ***When i do *solr query*:
> localhost:8080/solr3.6/OA/terms?terms.fl=title&terms.regex=phy.*
> fluids&terms.regex.flag=case_insensitive&terms.limit=10
>
> *Output* is 3rd title:
> *Physics of fluids*
>
> This is relevant output.
>
> ***But when i do *solr query*:
>
> localhost:8080/solr3.6/OA/terms?terms.fl=title&terms.regex=med.*
> phy.*&terms.regex.flag=case_insensitive&terms.limit=10
>
> *Output* are 2nd and 4th title:
>
> *Medical Engineering and Physics*
> *Medical Physics*
>
> This is irrelevant.I want only one result for this query i.e. *Medical
> Physics*
>
> *********************************Although i have changed my wild card
> characters to *[a-z0-9]** instead of *.** ,but than first query doesn't work
> as '*of*' is included in '*Physics of fluids*'.However Second query works
> fine .
>
> example of query is:
>
> localhost:8080/solr3.6/OA/terms?terms.fl=title&terms.regex=med[a-z0-9]*
> phy[a-z0-9]*&terms.regex.flag=case_insensitive&terms.limit=10
>
> This works fine,gives one output *Medical Physics*.
>
>
> If there is another way for searching using *Term Component* or without
> using it..Please suggest to neglect such stop words.
>
> Note:Term Component works only on string dataType field. :(
>
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Problem-using-Term-Component-in-solr-tp4077200.html
> Sent from the Solr - User mailing list archive at Nabble.com.