You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Hasnain <ha...@hotmail.com> on 2010/08/31 12:11:41 UTC
Alphanumeric wildcard search problem
I have gone through all the of the related posts, but could not find a proper
answer that works, so Im writing this post
Is there anyway of using wilcard searches on alphanumeric text like...R-1* ?
let me share relevent information
<fieldType name="textShoaib" class="solr.TextField"
positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.StandardTokenizerFactory"/> <!--This was
originally <tokenizer class="solr.WhitespaceTokenizerFactory"/> just playing
around-->
<filter class="solr.StopFilterFactory"
ignoreCase="true"
words="stopwords.txt"
enablePositionIncrements="true"
/>
<filter class="solr.WordDelimiterFilterFactory"
generateWordParts="0" generateNumberParts="0" catenateWords="0"
catenateNumbers="0" catenateAll="0" splitOnCaseChange="0"
preserveOriginal="1"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.EnglishPorterFilterFactory"
protected="protwords.txt"/>
<filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.StandardTokenizerFactory"/>
<!--This was originally <tokenizer
class="solr.WhitespaceTokenizerFactory"/>-->
<filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt"
ignoreCase="true" expand="true"/>
<filter class="solr.StopFilterFactory" ignoreCase="true"
words="stopwords.txt"/>
<filter class="solr.WordDelimiterFilterFactory"
generateWordParts="0" generateNumberParts="0" catenateWords="0"
catenateNumbers="0" catenateAll="0" splitOnCaseChange="0"
preserveOriginal="1"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.EnglishPorterFilterFactory"
protected="protwords.txt"/>
<filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
</analyzer>
</fieldType>
my requestHandler is...
<requestHandler name="standard2" class="solr.SearchHandler">
<!-- default values for query parameters -->
<lst name="defaults">
<str name="defType">dismax</str>
<str name="echoParams">explicit</str>
<str name="tie">0.6</str>
<str name="pf">name^2.3 mat_nr^0.4</str>
<str name="mm">0%</str>
<!--
<int name="rows">10</int>
<str name="fl">*</str>
<str name="version">2.1</str>
-->
</lst>
</requestHandler>
and also the field on which I want to apply searching on
<field name="mat_nr" type="textShoaib" indexed="true" stored="true"
omitNorms="true"/>
and the query Im using is
qt=standard2&q=R-1*
but this still doesnt work.
any suggestions on this?
thanks
--
View this message in context: http://lucene.472066.n3.nabble.com/Alphanumeric-wildcard-search-problem-tp1393332p1393332.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Alphanumeric wildcard search problem
Posted by Erick Erickson <er...@gmail.com>.
Ah, thanks. That reconciles our differing results.
Best
Erick
On Wed, Sep 8, 2010 at 2:58 AM, Hasnain <ha...@hotmail.com> wrote:
>
> The real problem was this tag
>
> <!-- field for the QueryParser to use when an explicit fieldname is absent
> -->
> <defaultSearchField>text</defaultSearchField>
>
> and I was quering like this q=r-1* instead of q=mat_nr:r-1*
> so whatever fieldType I use for mat_nr, it was using "text" fieldType which
> had WordDelimiterFilterFactory, hence I had to put space inorder to get it
> running.
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Alphanumeric-wildcard-search-problem-tp1393332p1437584.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
Re: Alphanumeric wildcard search problem
Posted by Hasnain <ha...@hotmail.com>.
The real problem was this tag
<!-- field for the QueryParser to use when an explicit fieldname is absent
-->
<defaultSearchField>text</defaultSearchField>
and I was quering like this q=r-1* instead of q=mat_nr:r-1*
so whatever fieldType I use for mat_nr, it was using "text" fieldType which
had WordDelimiterFilterFactory, hence I had to put space inorder to get it
running.
--
View this message in context: http://lucene.472066.n3.nabble.com/Alphanumeric-wildcard-search-problem-tp1393332p1437584.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Alphanumeric wildcard search problem
Posted by Erick Erickson <er...@gmail.com>.
Thanks for letting us know. What was the magic? I'm still unclear
what was different between my tests and your implementation, mysteries
like this make me nervous <G>..
Thanks
Erick
On Mon, Sep 6, 2010 at 5:45 PM, Hasnain <ha...@hotmail.com> wrote:
>
> Finally got it working, thanks for your help and support
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Alphanumeric-wildcard-search-problem-tp1393332p1429315.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
Re: Alphanumeric wildcard search problem
Posted by Hasnain <ha...@hotmail.com>.
Finally got it working, thanks for your help and support
--
View this message in context: http://lucene.472066.n3.nabble.com/Alphanumeric-wildcard-search-problem-tp1393332p1429315.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Alphanumeric wildcard search problem
Posted by Hasnain <ha...@hotmail.com>.
Hi Erik,
So I took your advice and started fresh with solr, got my self
latest copy of solr and started adding things gradually in configuration
files. Unfortunately, this still doesnot work. But, I realized that
searching for q=r-1* didnt return any results but when I query like this
q=r-(space)1*, solr returned perfect results.
when ever there is start of digits I have to put space before first digit
and it works perfect like dcp123* doesnot return no results but
dcp(space)123* works perfectly. Im using standard request handler and same
type i.e. textShoaib and Im absolutely sure im using the same schema.
Im sensing im very close to accomplish my task, any idea why solr is
behaving like that.
Is there anything I can do to get rid of this (space) issue before digits?
thanks
Erick Erickson wrote:
>
> All I can say is that I just tried it with the following definitions
> and it works as expected. That is:
> http://localhost:8983/solr/select/?q=eoe:R-1*&version=2.2&start=0&rows=10&indent=on
>
> <http://localhost:8983/solr/select/?q=eoe:R-1*&version=2.2&start=0&rows=10&indent=on>returns
> nothing (casing issue) and
>
> http://localhost:8983/solr/select/?q=eoe:r-1*&version=2.2&start=0&rows=10&indent=on<http://localhost:8983/solr/select/?q=eoe:R-1*&version=2.2&start=0&rows=10&indent=on>
>
> <http://localhost:8983/solr/select/?q=eoe:R-1*&version=2.2&start=0&rows=10&indent=on>returned
> 3 documents
>
> <fieldType name="textShoaib" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer type="index">
> <tokenizer class="solr.StandardTokenizerFactory"/> just playing
> <filter class="solr.StopFilterFactory"
> ignoreCase="true"
> words="stopwords.txt"
> enablePositionIncrements="true"
> />
> <filter class="solr.LowerCaseFilterFactory"/>
> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer
> class="solr.StandardTokenizerFactory"/>
> <filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt"
> ignoreCase="true" expand="true"/>
> <filter class="solr.StopFilterFactory" ignoreCase="true"
> words="stopwords.txt"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
> </analyzer>
> </fieldType>
>
> and
> <field name="eoe" type="textShoaib" indexed="true" />
>
> where the relevant documents had the following:
>
> <field name="eoe">R-1 and some stuff</field>
> <field name="eoe">r-1 really now</field>
> <field name="eoe">R-1</field>
>
> searching for r* and r-* also returned three documents.
> Note that using StandardAnalyzer and WhitespaceAnalyzer
> didn't make any difference.
>
> browsing the schema from the admin page shows the tokena
> r-1
> really
> some
> stuff
> now
>
> Are you totally and absolutely sure that you're using the schema you think
> you are?
> And you restarted SOLR after you made schema changes?
> And you re-indexed your data?
>
>
> I also tried it with the dismax query, and it works there too:
> http://localhost:8983/solr/select/?qt=standard2&q=r-1*&version=2.2&start=0&rows=10&indent=on&debugQuery=true
>
> returns 3 results, where standard2 is defined like this:
>
> <requestHandler name="standard2" class="solr.SearchHandler">
> <!-- default values for query parameters -->
> <lst name="defaults">
> <str name="defType">dismax</str>
> <str name="echoParams">explicit</str>
> <str name="tie">0.6</str>
> <str name="qf">eoe^2.3 mat_nr^0.4</str>
> <str name="mm">0%</str>
> <!--
> <int name="rows">10</int>
> <str name="fl">*</str>
> <str name="version">2.1</str>
> -->
> </lst>
>
>
> So in sum, I suspect that you've made some innocous-seeming
> change somewhere that you've forgotten about (easy to do when
> you're trying a zillion things and frustrated). So, here's what I'd
> recommend:
>
> back up and forget dismax for a while, just get it all working
> with the normal query parser and gradually add things back
> in until you either fail succeed all the way back to dismax.
>
> BTW, you probably want to specify dismax with the qf parameter,
> pf stands for "phrase fields" where qf stands for "query fields".
> pf worked for me, but I doubt that's what you're really after.
>
> This is all on SOLR 1.4.1 BTW.
>
> Best
> Erick
>
>
> On Thu, Sep 2, 2010 at 6:12 AM, Hasnain <ha...@hotmail.com> wrote:
>
>>
>> Erick,
>> I have checked with lowercasing, and yes there are Items by this
>> name.
>> Im not getting anywhere with this, tried many things and Im really
>> perplexed.
>>
>> any other suggestion?
>>
>>
>>
>>
>>
>> Oh dear. Wildcard queries aren't analyzed, so I suspect it's a casing
>> issue.
>>
>> Try two things:
>> 1> search for r-1*
>> 2> look in your index and be sure the actual terms are there as you
>> expect.
>>
>> HTH
>> Erick
>>
>>
>> --
>> View this message in context:
>> http://lucene.472066.n3.nabble.com/Alphanumeric-wildcard-search-problem-tp1393332p1405431.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
>
>
--
View this message in context: http://lucene.472066.n3.nabble.com/Alphanumeric-wildcard-search-problem-tp1393332p1427565.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Alphanumeric wildcard search problem
Posted by Erick Erickson <er...@gmail.com>.
All I can say is that I just tried it with the following definitions
and it works as expected. That is:
http://localhost:8983/solr/select/?q=eoe:R-1*&version=2.2&start=0&rows=10&indent=on
<http://localhost:8983/solr/select/?q=eoe:R-1*&version=2.2&start=0&rows=10&indent=on>returns
nothing (casing issue) and
http://localhost:8983/solr/select/?q=eoe:r-1*&version=2.2&start=0&rows=10&indent=on<http://localhost:8983/solr/select/?q=eoe:R-1*&version=2.2&start=0&rows=10&indent=on>
<http://localhost:8983/solr/select/?q=eoe:R-1*&version=2.2&start=0&rows=10&indent=on>returned
3 documents
<fieldType name="textShoaib" class="solr.TextField"
positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.StandardTokenizerFactory"/> just playing
<filter class="solr.StopFilterFactory"
ignoreCase="true"
words="stopwords.txt"
enablePositionIncrements="true"
/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer
class="solr.StandardTokenizerFactory"/>
<filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt"
ignoreCase="true" expand="true"/>
<filter class="solr.StopFilterFactory" ignoreCase="true"
words="stopwords.txt"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
</analyzer>
</fieldType>
and
<field name="eoe" type="textShoaib" indexed="true" />
where the relevant documents had the following:
<field name="eoe">R-1 and some stuff</field>
<field name="eoe">r-1 really now</field>
<field name="eoe">R-1</field>
searching for r* and r-* also returned three documents.
Note that using StandardAnalyzer and WhitespaceAnalyzer
didn't make any difference.
browsing the schema from the admin page shows the tokena
r-1
really
some
stuff
now
Are you totally and absolutely sure that you're using the schema you think
you are?
And you restarted SOLR after you made schema changes?
And you re-indexed your data?
I also tried it with the dismax query, and it works there too:
http://localhost:8983/solr/select/?qt=standard2&q=r-1*&version=2.2&start=0&rows=10&indent=on&debugQuery=true
returns 3 results, where standard2 is defined like this:
<requestHandler name="standard2" class="solr.SearchHandler">
<!-- default values for query parameters -->
<lst name="defaults">
<str name="defType">dismax</str>
<str name="echoParams">explicit</str>
<str name="tie">0.6</str>
<str name="qf">eoe^2.3 mat_nr^0.4</str>
<str name="mm">0%</str>
<!--
<int name="rows">10</int>
<str name="fl">*</str>
<str name="version">2.1</str>
-->
</lst>
So in sum, I suspect that you've made some innocous-seeming
change somewhere that you've forgotten about (easy to do when
you're trying a zillion things and frustrated). So, here's what I'd
recommend:
back up and forget dismax for a while, just get it all working
with the normal query parser and gradually add things back
in until you either fail succeed all the way back to dismax.
BTW, you probably want to specify dismax with the qf parameter,
pf stands for "phrase fields" where qf stands for "query fields".
pf worked for me, but I doubt that's what you're really after.
This is all on SOLR 1.4.1 BTW.
Best
Erick
On Thu, Sep 2, 2010 at 6:12 AM, Hasnain <ha...@hotmail.com> wrote:
>
> Erick,
> I have checked with lowercasing, and yes there are Items by this
> name.
> Im not getting anywhere with this, tried many things and Im really
> perplexed.
>
> any other suggestion?
>
>
>
>
>
> Oh dear. Wildcard queries aren't analyzed, so I suspect it's a casing
> issue.
>
> Try two things:
> 1> search for r-1*
> 2> look in your index and be sure the actual terms are there as you expect.
>
> HTH
> Erick
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Alphanumeric-wildcard-search-problem-tp1393332p1405431.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
Re: Alphanumeric wildcard search problem
Posted by Hasnain <ha...@hotmail.com>.
Erick,
I have checked with lowercasing, and yes there are Items by this
name.
Im not getting anywhere with this, tried many things and Im really
perplexed.
any other suggestion?
Oh dear. Wildcard queries aren't analyzed, so I suspect it's a casing issue.
Try two things:
1> search for r-1*
2> look in your index and be sure the actual terms are there as you expect.
HTH
Erick
--
View this message in context: http://lucene.472066.n3.nabble.com/Alphanumeric-wildcard-search-problem-tp1393332p1405431.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Alphanumeric wildcard search problem
Posted by Erick Erickson <er...@gmail.com>.
Oh dear. Wildcard queries aren't analyzed, so I suspect it's a casing issue.
Try two things:
1> search for r-1*
2> look in your index and be sure the actual terms are there as you expect.
HTH
Erick
On Wed, Sep 1, 2010 at 4:35 PM, Hasnain <ha...@hotmail.com> wrote:
>
> Thankyou for your suggestions
>
> when before removing the wordDelimiterFilterFactory, the results for q=R-*
> returned perfect results but not for q=R-1*, also after removing
> wordDelimiterFilterFactory, it didnt bring me results for q=R-*
>
> the results before removing wordDelimiterFilterFactory using debugQuery=on
> were
>
> <response>
> −
> <lst name="responseHeader">
> <int name="status">0</int>
> <int name="QTime">78</int>
> −
> <lst name="params">
> <str name="debugQuery">on</str>
> <str name="fl">mat_nr</str>
> <str name="q">R-1*</str>
> <str name="qt">standard2</str>
> </lst>
> </lst>
> <result name="response" numFound="0" start="0"/>
> −
> <lst name="debug">
> <str name="rawquerystring">R-1*</str>
> <str name="querystring">R-1*</str>
> −
> <str name="parsedquery">
> +DisjunctionMaxQuery((ext_quote_id:r-1*^0.4 | vendor_mat:r-1*^0.4 |
> description:r-1*^0.4 | prod_id:r-1*^0.4 | name:r-1*^2.3 | mat_nr:R-1*^0.4 |
> manufact_mat:r-1*^0.4)~0.6) ()
> </str>
> −
> <str name="parsedquery_toString">
> +(ext_quote_id:r-1*^0.4 | vendor_mat:r-1*^0.4 | description:r-1*^0.4 |
> prod_id:r-1*^0.4 | name:r-1*^2.3 | mat_nr:R-1*^0.4 |
> manufact_mat:r-1*^0.4)~0.6 ()
> </str>
> <lst name="explain"/>
> <str name="QParser">DisMaxQParser</str>
> <null name="altquerystring"/>
> <null name="boostfuncs"/>
> −
> <lst name="timing">
> <double name="time">31.0</double>
> −
> <lst name="prepare">
> <double name="time">15.0</double>
> −
> <lst name="org.apache.solr.handler.component.QueryComponent">
> <double name="time">15.0</double>
> </lst>
> −
> <lst name="org.apache.solr.handler.component.FacetComponent">
> <double name="time">0.0</double>
> </lst>
> −
> <lst name="org.apache.solr.handler.component.MoreLikeThisComponent">
> <double name="time">0.0</double>
> </lst>
> −
> <lst name="org.apache.solr.handler.component.HighlightComponent">
> <double name="time">0.0</double>
> </lst>
> −
> <lst name="org.apache.solr.handler.component.StatsComponent">
> <double name="time">0.0</double>
> </lst>
> −
> <lst name="org.apache.solr.handler.component.DebugComponent">
> <double name="time">0.0</double>
> </lst>
> </lst>
> −
> <lst name="process">
> <double name="time">16.0</double>
> −
> <lst name="org.apache.solr.handler.component.QueryComponent">
> <double name="time">16.0</double>
> </lst>
> −
> <lst name="org.apache.solr.handler.component.FacetComponent">
> <double name="time">0.0</double>
> </lst>
> −
> <lst name="org.apache.solr.handler.component.MoreLikeThisComponent">
> <double name="time">0.0</double>
> </lst>
> −
> <lst name="org.apache.solr.handler.component.HighlightComponent">
> <double name="time">0.0</double>
> </lst>
> −
> <lst name="org.apache.solr.handler.component.StatsComponent">
> <double name="time">0.0</double>
> </lst>
> −
> <lst name="org.apache.solr.handler.component.DebugComponent">
> <double name="time">0.0</double>
> </lst>
> </lst>
> </lst>
> </lst>
> </response>
>
> and after removing wordDelimiterFilterFactory
>
> <response>
> −
> <lst name="responseHeader">
> <int name="status">0</int>
> <int name="QTime">78</int>
> −
> <lst name="params">
> <str name="debugQuery">on</str>
> <str name="fl">mat_nr</str>
> <str name="q">R-1*</str>
> <str name="qt">standard2</str>
> </lst>
> </lst>
> <result name="response" numFound="0" start="0"/>
> −
> <lst name="debug">
> <str name="rawquerystring">R-1*</str>
> <str name="querystring">R-1*</str>
> −
> <str name="parsedquery">
> +DisjunctionMaxQuery((ext_quote_id:r-1*^0.4 | vendor_mat:r-1*^0.4 |
> description:r-1*^0.4 | prod_id:r-1*^0.4 | name:r-1*^2.3 | mat_nr:R-1*^0.4 |
> manufact_mat:r-1*^0.4)~0.6) ()
> </str>
> −
> <str name="parsedquery_toString">
> +(ext_quote_id:r-1*^0.4 | vendor_mat:r-1*^0.4 | description:r-1*^0.4 |
> prod_id:r-1*^0.4 | name:r-1*^2.3 | mat_nr:R-1*^0.4 |
> manufact_mat:r-1*^0.4)~0.6 ()
> </str>
> <lst name="explain"/>
> <str name="QParser">DisMaxQParser</str>
> <null name="altquerystring"/>
> <null name="boostfuncs"/>
> −
> <lst name="timing">
> <double name="time">31.0</double>
> −
> <lst name="prepare">
> <double name="time">15.0</double>
> −
> <lst name="org.apache.solr.handler.component.QueryComponent">
> <double name="time">15.0</double>
> </lst>
> −
> <lst name="org.apache.solr.handler.component.FacetComponent">
> <double name="time">0.0</double>
> </lst>
> −
> <lst name="org.apache.solr.handler.component.MoreLikeThisComponent">
> <double name="time">0.0</double>
> </lst>
> −
> <lst name="org.apache.solr.handler.component.HighlightComponent">
> <double name="time">0.0</double>
> </lst>
> −
> <lst name="org.apache.solr.handler.component.StatsComponent">
> <double name="time">0.0</double>
> </lst>
> −
> <lst name="org.apache.solr.handler.component.DebugComponent">
> <double name="time">0.0</double>
> </lst>
> </lst>
> −
> <lst name="process">
> <double name="time">16.0</double>
> −
> <lst name="org.apache.solr.handler.component.QueryComponent">
> <double name="time">16.0</double>
> </lst>
> −
> <lst name="org.apache.solr.handler.component.FacetComponent">
> <double name="time">0.0</double>
> </lst>
> −
> <lst name="org.apache.solr.handler.component.MoreLikeThisComponent">
> <double name="time">0.0</double>
> </lst>
> −
> <lst name="org.apache.solr.handler.component.HighlightComponent">
> <double name="time">0.0</double>
> </lst>
> −
> <lst name="org.apache.solr.handler.component.StatsComponent">
> <double name="time">0.0</double>
> </lst>
> −
> <lst name="org.apache.solr.handler.component.DebugComponent">
> <double name="time">0.0</double>
> </lst>
> </lst>
> </lst>
> </lst>
> </response>
>
> also at first the wordDelimiterFilterFactory used was this
> <filter class="solr.WordDelimiterFilterFactory" generateWordParts="1"
> generateNumberParts="1" catenateWords="1" catenateNumbers="1"
> catenateAll="0" splitOnCaseChange="1"/>
> <filter class="solr.LowerCaseFilterFactory"/>
>
> before removing wordDelimiterFilterFactory, solr admin showed
>
> Index Analyzer
> org.apache.solr.analysis.WhitespaceTokenizerFactory {}
> term position 1
> term text R-1110
> term type word
> source start,end 0,6
> payload
> org.apache.solr.analysis.StopFilterFactory {words=stopwords.txt,
> ignoreCase=true, enablePositionIncrements=true}
> term position 1
> term text R-1110
> term type word
> source start,end 0,6
> payload
> org.apache.solr.analysis.WordDelimiterFilterFactory {splitOnCaseChange=1,
> generateNumberParts=1, catenateWords=1, generateWordParts=1, catenateAll=0,
> catenateNumbers=1}
> term position 1 2
> term text R 1110
> term type word word
> source start,end 0,1 2,6
> payload
> org.apache.solr.analysis.LowerCaseFilterFactory {}
> term position 1 2
> term text r 1110
> term type word word
> source start,end 0,1 2,6
> payload
> org.apache.solr.analysis.EnglishPorterFilterFactory
> {protected=protwords.txt}
> term position 1 2
> term text r 1110
> term type word word
> source start,end 0,1 2,6
> payload
> org.apache.solr.analysis.RemoveDuplicatesTokenFilterFactory {}
> term position 1 2
> term text r 1110
> term type word word
> source start,end 0,1 2,6
> payload
>
>
>
> also after removing wordDelimiterFilterFactory,solr admin looks like this
>
> Index Analyzer
> org.apache.solr.analysis.WhitespaceTokenizerFactory {}
> term position 1
> term text R-1110
> term type word
> source start,end 0,6
> payload
> org.apache.solr.analysis.StopFilterFactory {words=stopwords.txt,
> ignoreCase=true, enablePositionIncrements=true}
> term position 1
> term text R-1110
> term type word
> source start,end 0,6
> payload
> org.apache.solr.analysis.EnglishPorterFilterFactory
> {protected=protwords.txt}
> term position 1
> term text R-1110
> term type word
> source start,end 0,6
> payload
> org.apache.solr.analysis.RemoveDuplicatesTokenFilterFactory {}
> term position 1
> term text R-1110
> term type word
> source start,end 0,6
> payload
>
>
> any suggestions?
>
> thankyou
>
>
> Erick Erickson wrote:
> >
> > Really look at the analysis page in solr admin for how your
> > analyzer chain handles things, or you'll spend time until you're
> > really old having trouble :).
> >
> > Here's what I see on a quick scan:
> >
> >> StandardTokenizer tries to, among other things, preserve
> > email addresses. The kinds of strings you're working with may
> > trip something up here.
> >
> >> Remove WordDelimiterFactory altogether. The point of WDF
> > is to break words apart at transitions.
> >
> >> Remove EnglishPorterFilterFactory too. What the effect
> > of applying an algorithmic stemming process to words like
> > you're interested in is...er...not obvious.
> >
> > All that said, I took a quick at the analysis page with your definition
> > and nothing jumped out at me. Are you sure that:
> >> you're getting to the request handler you think? What does adding
> > &debugQuery=on show?
> >> you've indexed the data after you've made the changes you outlined
> above?
> > The SOLR
> > admin page can help here, especially the [full interface] link, with
> debug
> > info on.
> >
> > If nothing shows up, can you post the results of &debugQuery=on?
> >
> > Best
> > Erick
> >
> > On Tue, Aug 31, 2010 at 6:11 AM, Hasnain <ha...@hotmail.com> wrote:
> >
> >>
> >> I have gone through all the of the related posts, but could not find a
> >> proper
> >> answer that works, so Im writing this post
> >>
> >> Is there anyway of using wilcard searches on alphanumeric text
> >> like...R-1*
> >> ?
> >>
> >> let me share relevent information
> >>
> >>
> >> <fieldType name="textShoaib" class="solr.TextField"
> >> positionIncrementGap="100">
> >> <analyzer type="index">
> >> <tokenizer class="solr.StandardTokenizerFactory"/> <!--This was
> >> originally <tokenizer class="solr.WhitespaceTokenizerFactory"/> just
> >> playing
> >> around-->
> >> <filter class="solr.StopFilterFactory"
> >> ignoreCase="true"
> >> words="stopwords.txt"
> >> enablePositionIncrements="true"
> >> />
> >> <filter class="solr.WordDelimiterFilterFactory"
> >> generateWordParts="0" generateNumberParts="0" catenateWords="0"
> >> catenateNumbers="0" catenateAll="0" splitOnCaseChange="0"
> >> preserveOriginal="1"/>
> >> <filter class="solr.LowerCaseFilterFactory"/>
> >> <filter class="solr.EnglishPorterFilterFactory"
> >> protected="protwords.txt"/>
> >> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
> >> </analyzer>
> >> <analyzer type="query">
> >> <tokenizer class="solr.StandardTokenizerFactory"/>
> >> <!--This was originally <tokenizer
> >> class="solr.WhitespaceTokenizerFactory"/>-->
> >> <filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt"
> >> ignoreCase="true" expand="true"/>
> >> <filter class="solr.StopFilterFactory" ignoreCase="true"
> >> words="stopwords.txt"/>
> >> <filter class="solr.WordDelimiterFilterFactory"
> >> generateWordParts="0" generateNumberParts="0" catenateWords="0"
> >> catenateNumbers="0" catenateAll="0" splitOnCaseChange="0"
> >> preserveOriginal="1"/>
> >> <filter class="solr.LowerCaseFilterFactory"/>
> >> <filter class="solr.EnglishPorterFilterFactory"
> >> protected="protwords.txt"/>
> >> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
> >> </analyzer>
> >> </fieldType>
> >>
> >>
> >>
> >>
> >> my requestHandler is...
> >>
> >>
> >>
> >>
> >>
> >> <requestHandler name="standard2" class="solr.SearchHandler">
> >> <!-- default values for query parameters -->
> >> <lst name="defaults">
> >> <str name="defType">dismax</str>
> >> <str name="echoParams">explicit</str>
> >> <str name="tie">0.6</str>
> >> <str name="pf">name^2.3 mat_nr^0.4</str>
> >> <str name="mm">0%</str>
> >> <!--
> >> <int name="rows">10</int>
> >> <str name="fl">*</str>
> >> <str name="version">2.1</str>
> >> -->
> >> </lst>
> >>
> >> </requestHandler>
> >>
> >>
> >>
> >> and also the field on which I want to apply searching on
> >>
> >>
> >>
> >> <field name="mat_nr" type="textShoaib" indexed="true" stored="true"
> >> omitNorms="true"/>
> >>
> >>
> >>
> >> and the query Im using is
> >>
> >>
> >>
> >> qt=standard2&q=R-1*
> >>
> >>
> >>
> >> but this still doesnt work.
> >>
> >>
> >> any suggestions on this?
> >>
> >> thanks
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >> --
> >> View this message in context:
> >>
> http://lucene.472066.n3.nabble.com/Alphanumeric-wildcard-search-problem-tp1393332p1393332.html
> >> Sent from the Solr - User mailing list archive at Nabble.com.
> >>
> >
> >
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Alphanumeric-wildcard-search-problem-tp1393332p1402772.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
Re: Alphanumeric wildcard search problem
Posted by Hasnain <ha...@hotmail.com>.
Thankyou for your suggestions
when before removing the wordDelimiterFilterFactory, the results for q=R-*
returned perfect results but not for q=R-1*, also after removing
wordDelimiterFilterFactory, it didnt bring me results for q=R-*
the results before removing wordDelimiterFilterFactory using debugQuery=on
were
<response>
−
<lst name="responseHeader">
<int name="status">0</int>
<int name="QTime">78</int>
−
<lst name="params">
<str name="debugQuery">on</str>
<str name="fl">mat_nr</str>
<str name="q">R-1*</str>
<str name="qt">standard2</str>
</lst>
</lst>
<result name="response" numFound="0" start="0"/>
−
<lst name="debug">
<str name="rawquerystring">R-1*</str>
<str name="querystring">R-1*</str>
−
<str name="parsedquery">
+DisjunctionMaxQuery((ext_quote_id:r-1*^0.4 | vendor_mat:r-1*^0.4 |
description:r-1*^0.4 | prod_id:r-1*^0.4 | name:r-1*^2.3 | mat_nr:R-1*^0.4 |
manufact_mat:r-1*^0.4)~0.6) ()
</str>
−
<str name="parsedquery_toString">
+(ext_quote_id:r-1*^0.4 | vendor_mat:r-1*^0.4 | description:r-1*^0.4 |
prod_id:r-1*^0.4 | name:r-1*^2.3 | mat_nr:R-1*^0.4 |
manufact_mat:r-1*^0.4)~0.6 ()
</str>
<lst name="explain"/>
<str name="QParser">DisMaxQParser</str>
<null name="altquerystring"/>
<null name="boostfuncs"/>
−
<lst name="timing">
<double name="time">31.0</double>
−
<lst name="prepare">
<double name="time">15.0</double>
−
<lst name="org.apache.solr.handler.component.QueryComponent">
<double name="time">15.0</double>
</lst>
−
<lst name="org.apache.solr.handler.component.FacetComponent">
<double name="time">0.0</double>
</lst>
−
<lst name="org.apache.solr.handler.component.MoreLikeThisComponent">
<double name="time">0.0</double>
</lst>
−
<lst name="org.apache.solr.handler.component.HighlightComponent">
<double name="time">0.0</double>
</lst>
−
<lst name="org.apache.solr.handler.component.StatsComponent">
<double name="time">0.0</double>
</lst>
−
<lst name="org.apache.solr.handler.component.DebugComponent">
<double name="time">0.0</double>
</lst>
</lst>
−
<lst name="process">
<double name="time">16.0</double>
−
<lst name="org.apache.solr.handler.component.QueryComponent">
<double name="time">16.0</double>
</lst>
−
<lst name="org.apache.solr.handler.component.FacetComponent">
<double name="time">0.0</double>
</lst>
−
<lst name="org.apache.solr.handler.component.MoreLikeThisComponent">
<double name="time">0.0</double>
</lst>
−
<lst name="org.apache.solr.handler.component.HighlightComponent">
<double name="time">0.0</double>
</lst>
−
<lst name="org.apache.solr.handler.component.StatsComponent">
<double name="time">0.0</double>
</lst>
−
<lst name="org.apache.solr.handler.component.DebugComponent">
<double name="time">0.0</double>
</lst>
</lst>
</lst>
</lst>
</response>
and after removing wordDelimiterFilterFactory
<response>
−
<lst name="responseHeader">
<int name="status">0</int>
<int name="QTime">78</int>
−
<lst name="params">
<str name="debugQuery">on</str>
<str name="fl">mat_nr</str>
<str name="q">R-1*</str>
<str name="qt">standard2</str>
</lst>
</lst>
<result name="response" numFound="0" start="0"/>
−
<lst name="debug">
<str name="rawquerystring">R-1*</str>
<str name="querystring">R-1*</str>
−
<str name="parsedquery">
+DisjunctionMaxQuery((ext_quote_id:r-1*^0.4 | vendor_mat:r-1*^0.4 |
description:r-1*^0.4 | prod_id:r-1*^0.4 | name:r-1*^2.3 | mat_nr:R-1*^0.4 |
manufact_mat:r-1*^0.4)~0.6) ()
</str>
−
<str name="parsedquery_toString">
+(ext_quote_id:r-1*^0.4 | vendor_mat:r-1*^0.4 | description:r-1*^0.4 |
prod_id:r-1*^0.4 | name:r-1*^2.3 | mat_nr:R-1*^0.4 |
manufact_mat:r-1*^0.4)~0.6 ()
</str>
<lst name="explain"/>
<str name="QParser">DisMaxQParser</str>
<null name="altquerystring"/>
<null name="boostfuncs"/>
−
<lst name="timing">
<double name="time">31.0</double>
−
<lst name="prepare">
<double name="time">15.0</double>
−
<lst name="org.apache.solr.handler.component.QueryComponent">
<double name="time">15.0</double>
</lst>
−
<lst name="org.apache.solr.handler.component.FacetComponent">
<double name="time">0.0</double>
</lst>
−
<lst name="org.apache.solr.handler.component.MoreLikeThisComponent">
<double name="time">0.0</double>
</lst>
−
<lst name="org.apache.solr.handler.component.HighlightComponent">
<double name="time">0.0</double>
</lst>
−
<lst name="org.apache.solr.handler.component.StatsComponent">
<double name="time">0.0</double>
</lst>
−
<lst name="org.apache.solr.handler.component.DebugComponent">
<double name="time">0.0</double>
</lst>
</lst>
−
<lst name="process">
<double name="time">16.0</double>
−
<lst name="org.apache.solr.handler.component.QueryComponent">
<double name="time">16.0</double>
</lst>
−
<lst name="org.apache.solr.handler.component.FacetComponent">
<double name="time">0.0</double>
</lst>
−
<lst name="org.apache.solr.handler.component.MoreLikeThisComponent">
<double name="time">0.0</double>
</lst>
−
<lst name="org.apache.solr.handler.component.HighlightComponent">
<double name="time">0.0</double>
</lst>
−
<lst name="org.apache.solr.handler.component.StatsComponent">
<double name="time">0.0</double>
</lst>
−
<lst name="org.apache.solr.handler.component.DebugComponent">
<double name="time">0.0</double>
</lst>
</lst>
</lst>
</lst>
</response>
also at first the wordDelimiterFilterFactory used was this
<filter class="solr.WordDelimiterFilterFactory" generateWordParts="1"
generateNumberParts="1" catenateWords="1" catenateNumbers="1"
catenateAll="0" splitOnCaseChange="1"/>
<filter class="solr.LowerCaseFilterFactory"/>
before removing wordDelimiterFilterFactory, solr admin showed
Index Analyzer
org.apache.solr.analysis.WhitespaceTokenizerFactory {}
term position 1
term text R-1110
term type word
source start,end 0,6
payload
org.apache.solr.analysis.StopFilterFactory {words=stopwords.txt,
ignoreCase=true, enablePositionIncrements=true}
term position 1
term text R-1110
term type word
source start,end 0,6
payload
org.apache.solr.analysis.WordDelimiterFilterFactory {splitOnCaseChange=1,
generateNumberParts=1, catenateWords=1, generateWordParts=1, catenateAll=0,
catenateNumbers=1}
term position 1 2
term text R 1110
term type word word
source start,end 0,1 2,6
payload
org.apache.solr.analysis.LowerCaseFilterFactory {}
term position 1 2
term text r 1110
term type word word
source start,end 0,1 2,6
payload
org.apache.solr.analysis.EnglishPorterFilterFactory
{protected=protwords.txt}
term position 1 2
term text r 1110
term type word word
source start,end 0,1 2,6
payload
org.apache.solr.analysis.RemoveDuplicatesTokenFilterFactory {}
term position 1 2
term text r 1110
term type word word
source start,end 0,1 2,6
payload
also after removing wordDelimiterFilterFactory,solr admin looks like this
Index Analyzer
org.apache.solr.analysis.WhitespaceTokenizerFactory {}
term position 1
term text R-1110
term type word
source start,end 0,6
payload
org.apache.solr.analysis.StopFilterFactory {words=stopwords.txt,
ignoreCase=true, enablePositionIncrements=true}
term position 1
term text R-1110
term type word
source start,end 0,6
payload
org.apache.solr.analysis.EnglishPorterFilterFactory
{protected=protwords.txt}
term position 1
term text R-1110
term type word
source start,end 0,6
payload
org.apache.solr.analysis.RemoveDuplicatesTokenFilterFactory {}
term position 1
term text R-1110
term type word
source start,end 0,6
payload
any suggestions?
thankyou
Erick Erickson wrote:
>
> Really look at the analysis page in solr admin for how your
> analyzer chain handles things, or you'll spend time until you're
> really old having trouble :).
>
> Here's what I see on a quick scan:
>
>> StandardTokenizer tries to, among other things, preserve
> email addresses. The kinds of strings you're working with may
> trip something up here.
>
>> Remove WordDelimiterFactory altogether. The point of WDF
> is to break words apart at transitions.
>
>> Remove EnglishPorterFilterFactory too. What the effect
> of applying an algorithmic stemming process to words like
> you're interested in is...er...not obvious.
>
> All that said, I took a quick at the analysis page with your definition
> and nothing jumped out at me. Are you sure that:
>> you're getting to the request handler you think? What does adding
> &debugQuery=on show?
>> you've indexed the data after you've made the changes you outlined above?
> The SOLR
> admin page can help here, especially the [full interface] link, with debug
> info on.
>
> If nothing shows up, can you post the results of &debugQuery=on?
>
> Best
> Erick
>
> On Tue, Aug 31, 2010 at 6:11 AM, Hasnain <ha...@hotmail.com> wrote:
>
>>
>> I have gone through all the of the related posts, but could not find a
>> proper
>> answer that works, so Im writing this post
>>
>> Is there anyway of using wilcard searches on alphanumeric text
>> like...R-1*
>> ?
>>
>> let me share relevent information
>>
>>
>> <fieldType name="textShoaib" class="solr.TextField"
>> positionIncrementGap="100">
>> <analyzer type="index">
>> <tokenizer class="solr.StandardTokenizerFactory"/> <!--This was
>> originally <tokenizer class="solr.WhitespaceTokenizerFactory"/> just
>> playing
>> around-->
>> <filter class="solr.StopFilterFactory"
>> ignoreCase="true"
>> words="stopwords.txt"
>> enablePositionIncrements="true"
>> />
>> <filter class="solr.WordDelimiterFilterFactory"
>> generateWordParts="0" generateNumberParts="0" catenateWords="0"
>> catenateNumbers="0" catenateAll="0" splitOnCaseChange="0"
>> preserveOriginal="1"/>
>> <filter class="solr.LowerCaseFilterFactory"/>
>> <filter class="solr.EnglishPorterFilterFactory"
>> protected="protwords.txt"/>
>> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
>> </analyzer>
>> <analyzer type="query">
>> <tokenizer class="solr.StandardTokenizerFactory"/>
>> <!--This was originally <tokenizer
>> class="solr.WhitespaceTokenizerFactory"/>-->
>> <filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt"
>> ignoreCase="true" expand="true"/>
>> <filter class="solr.StopFilterFactory" ignoreCase="true"
>> words="stopwords.txt"/>
>> <filter class="solr.WordDelimiterFilterFactory"
>> generateWordParts="0" generateNumberParts="0" catenateWords="0"
>> catenateNumbers="0" catenateAll="0" splitOnCaseChange="0"
>> preserveOriginal="1"/>
>> <filter class="solr.LowerCaseFilterFactory"/>
>> <filter class="solr.EnglishPorterFilterFactory"
>> protected="protwords.txt"/>
>> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
>> </analyzer>
>> </fieldType>
>>
>>
>>
>>
>> my requestHandler is...
>>
>>
>>
>>
>>
>> <requestHandler name="standard2" class="solr.SearchHandler">
>> <!-- default values for query parameters -->
>> <lst name="defaults">
>> <str name="defType">dismax</str>
>> <str name="echoParams">explicit</str>
>> <str name="tie">0.6</str>
>> <str name="pf">name^2.3 mat_nr^0.4</str>
>> <str name="mm">0%</str>
>> <!--
>> <int name="rows">10</int>
>> <str name="fl">*</str>
>> <str name="version">2.1</str>
>> -->
>> </lst>
>>
>> </requestHandler>
>>
>>
>>
>> and also the field on which I want to apply searching on
>>
>>
>>
>> <field name="mat_nr" type="textShoaib" indexed="true" stored="true"
>> omitNorms="true"/>
>>
>>
>>
>> and the query Im using is
>>
>>
>>
>> qt=standard2&q=R-1*
>>
>>
>>
>> but this still doesnt work.
>>
>>
>> any suggestions on this?
>>
>> thanks
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> --
>> View this message in context:
>> http://lucene.472066.n3.nabble.com/Alphanumeric-wildcard-search-problem-tp1393332p1393332.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
>
>
--
View this message in context: http://lucene.472066.n3.nabble.com/Alphanumeric-wildcard-search-problem-tp1393332p1402772.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Alphanumeric wildcard search problem
Posted by Erick Erickson <er...@gmail.com>.
Really look at the analysis page in solr admin for how your
analyzer chain handles things, or you'll spend time until you're
really old having trouble :).
Here's what I see on a quick scan:
> StandardTokenizer tries to, among other things, preserve
email addresses. The kinds of strings you're working with may
trip something up here.
> Remove WordDelimiterFactory altogether. The point of WDF
is to break words apart at transitions.
> Remove EnglishPorterFilterFactory too. What the effect
of applying an algorithmic stemming process to words like
you're interested in is...er...not obvious.
All that said, I took a quick at the analysis page with your definition
and nothing jumped out at me. Are you sure that:
> you're getting to the request handler you think? What does adding
&debugQuery=on show?
> you've indexed the data after you've made the changes you outlined above?
The SOLR
admin page can help here, especially the [full interface] link, with debug
info on.
If nothing shows up, can you post the results of &debugQuery=on?
Best
Erick
On Tue, Aug 31, 2010 at 6:11 AM, Hasnain <ha...@hotmail.com> wrote:
>
> I have gone through all the of the related posts, but could not find a
> proper
> answer that works, so Im writing this post
>
> Is there anyway of using wilcard searches on alphanumeric text like...R-1*
> ?
>
> let me share relevent information
>
>
> <fieldType name="textShoaib" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer type="index">
> <tokenizer class="solr.StandardTokenizerFactory"/> <!--This was
> originally <tokenizer class="solr.WhitespaceTokenizerFactory"/> just
> playing
> around-->
> <filter class="solr.StopFilterFactory"
> ignoreCase="true"
> words="stopwords.txt"
> enablePositionIncrements="true"
> />
> <filter class="solr.WordDelimiterFilterFactory"
> generateWordParts="0" generateNumberParts="0" catenateWords="0"
> catenateNumbers="0" catenateAll="0" splitOnCaseChange="0"
> preserveOriginal="1"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> <filter class="solr.EnglishPorterFilterFactory"
> protected="protwords.txt"/>
> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.StandardTokenizerFactory"/>
> <!--This was originally <tokenizer
> class="solr.WhitespaceTokenizerFactory"/>-->
> <filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt"
> ignoreCase="true" expand="true"/>
> <filter class="solr.StopFilterFactory" ignoreCase="true"
> words="stopwords.txt"/>
> <filter class="solr.WordDelimiterFilterFactory"
> generateWordParts="0" generateNumberParts="0" catenateWords="0"
> catenateNumbers="0" catenateAll="0" splitOnCaseChange="0"
> preserveOriginal="1"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> <filter class="solr.EnglishPorterFilterFactory"
> protected="protwords.txt"/>
> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
> </analyzer>
> </fieldType>
>
>
>
>
> my requestHandler is...
>
>
>
>
>
> <requestHandler name="standard2" class="solr.SearchHandler">
> <!-- default values for query parameters -->
> <lst name="defaults">
> <str name="defType">dismax</str>
> <str name="echoParams">explicit</str>
> <str name="tie">0.6</str>
> <str name="pf">name^2.3 mat_nr^0.4</str>
> <str name="mm">0%</str>
> <!--
> <int name="rows">10</int>
> <str name="fl">*</str>
> <str name="version">2.1</str>
> -->
> </lst>
>
> </requestHandler>
>
>
>
> and also the field on which I want to apply searching on
>
>
>
> <field name="mat_nr" type="textShoaib" indexed="true" stored="true"
> omitNorms="true"/>
>
>
>
> and the query Im using is
>
>
>
> qt=standard2&q=R-1*
>
>
>
> but this still doesnt work.
>
>
> any suggestions on this?
>
> thanks
>
>
>
>
>
>
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Alphanumeric-wildcard-search-problem-tp1393332p1393332.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>