You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Tejas Bhanushali <co...@gmail.com> on 2016/04/15 19:48:59 UTC
Getting duplicate output while doing auto suggestion based on
multiple filed using copy filed in solr 5.5
Hi Team,
Im getting the duplicate result when i do auto suggestion based on multiple
filed by using copy filed . i have below table configuration .
Segment -- have multiple category -- have multiple sub category -- have
multiple products.
suggestion are given based on
segment name, category name, sub category name and product name.
below is output .
---------------
{
"responseHeader": {
"status": 0,
"QTime": 1375
},
"command": "build",
"suggest": {
"mySuggester": {
"Fruit": {
"numFound": 10,
"suggestions": [
{
"term": "<b>Fruit</b>s & Vegetables",
"weight": 1000,
"payload": ""
},
{
"term": "<b>Fruit</b>s & Vegetables",
"weight": 1000,
"payload": ""
},
{
"term": "<b>Fruit</b>s & Vegetables",
"weight": 980,
"payload": ""
},
{
"term": "<b>Fruit</b>s & Vegetables",
"weight": 980,
"payload": ""
},
{
"term": "<b>Fruit</b>s & Vegetables",
"weight": 800,
"payload": ""
},
{
"term": "<b>Fruit</b>s & Vegetables",
"weight": 588,
"payload": ""
},
{
"term": "Cut <b>Fruit</b>s",
"weight": 456,
"payload": ""
},
{
"term": "<b>Fruit</b>s",
"weight": 456,
"payload": ""
},
{
"term": "<b>Fruit</b>s & Vegetables",
"weight": 456,
"payload": ""
},
{
"term": "<b>Fruit</b>s",
"weight": 456,
"payload": ""
}
]
}
}
}
}
--
Thanks & Regards,
Tejas Bhanushali
Re: Getting duplicate output while doing auto suggestion based on
multiple filed using copy filed in solr 5.5
Posted by Tejas Bhanushali <co...@gmail.com>.
Hi Team ,
Please follow the below steps to reproduce this issue.for that you to
change the existing config file under the
directory solr-5.5.0\server\solr\configsets\sample_techproducts_configs\conf
.
Note:- if you see the config i have used the "AnalyzingInfixLookupFactory"
instead of "FuzzyLookupFactory" for infix search .
Step 1:- First replace the existing "Suggest" component in solrconfig.xml
as mention below.
<searchComponent name="suggest" class="solr.SuggestComponent">
<lst name="suggester">
<str name="name">mySuggester</str>
<!--<str name="lookupImpl">FuzzyLookupFactory</str>
<str name="storeDir">suggester_fuzzy_dir</str>-->
<str name="lookupImpl">AnalyzingInfixLookupFactory</str>
<str name="indexPath">suggester_infix_dir</str>
<str name="dictionaryImpl">DocumentDictionaryFactory</str>
<str name="field">suggest_text</str>
<str name="weightField">price</str>
<str name="suggestAnalyzerFieldType">text_auto</str>
<str name="queryAnalyzerFieldType">text_auto</str>
<str name="buildOnStartup">false</str>
<str name="buildOnCommit">false</str>
</lst>
</searchComponent>
<requestHandler name="/suggest" class="solr.SearchHandler"
startup="lazy" >
<lst name="defaults">
<str name="suggest">true</str>
<str name="suggest.count">10</str>
<!--<str name="suggest.dictionary">mySuggester</str>-->
<str name="suggest.onlyMorePopular">true</str>
<str name="suggest.collate">true</str>
</lst>
<arr name="components">
<str>suggest</str>
</arr>
</requestHandler>
Step2:- Add new field into managed-schema for auto suggetion .
<field name="suggest_text" type="text_auto" indexed="true"
stored="true" multiValued="true"/>
Step3:- i have provided suggestion based on multiple fields i.e. on
cat,name and manu
<copyField source="cat" dest="suggest_text"/>
<copyField source="name" dest="suggest_text"/>
<copyField source="manu" dest="suggest_text"/>
Step4:- add below field type into solrconfig.xml. please note that i have
used NgramTokenizer to provide suggetion based on user type character into
search box.
eg. suppose user type "compu" then it should get suggetion like
"electronics and computer1"
<fieldType name="text_auto" class="solr.TextField"
positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.NGramTokenizerFactory" minGramSize="2"
maxGramSize="30"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.EdgeNGramTokenizerFactory" minGramSize="2"
maxGramSize="10"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
Step 5:- run the solr by using existing techproducts example
bin\solr -e techproducts
Step5:- Run the below command to build the schema
curl "http://localhost:8983/solr/techproducts/suggest?suggest.dic
tionary=mySuggester&suggest=true&suggest.build=true&wt=json"
Step6:- finally run the url like:-
curl "http://localhost:8983/solr/techproducts/suggest?wt=json&ind
ent=true&suggest.dictionary=mySuggester&suggest=true&avoidDuplicate=0&suggest.q=
electro"
Step 7:- you will get below response:-
{
"responseHeader":{
"status":0,
"QTime":15},
"suggest":{"mySuggester":{
"elec":{
"numFound":10,
"suggestions":[{
"term":"<b>el</b>ectronics and computer1",
"weight":2199,
"payload":""},
{
"term":"<b>el</b>ectronics",
"weight":649,
"payload":""},
{
"term":"<b>el</b>ectronics",
"weight":479,
"payload":""},
{
"term":"<b>el</b>ectronics",
"weight":399,
"payload":""},
{
"term":"<b>el</b>ectronics",
"weight":350,
"payload":""},
{
"term":"<b>el</b>ectronics",
"weight":329,
"payload":""},
{
"term":"<b>el</b>ectronics and stuff2",
"weight":279,
"payload":""},
{
"term":"<b>el</b>ectronics",
"weight":185,
"payload":""},
{
"term":"<b>el</b>ectronics",
"weight":179,
"payload":""},
{
"term":"Samsung Electronics Co. Ltd.",
"weight":92,
"payload":""}]}}}}
if you see the response then you will find the word "electronics" getting
repeating by 7 times.
PFA file for your reference if you unable to config as i mention above.
Please advice how to resolve this issue .
Thanks & Regards,
Tejas
On Mon, Apr 18, 2016 at 11:12 PM, Tejas Bhanushali <
contact.tejasbhanushali@gmail.com> wrote:
> HI Team,
>
> I tried to run the same example as suggested by Chris Hostetter and i get
> to know it's working fine for single field, but my requirement is it should
> suggest based on multiple fields .i.e not only on "cat" field but it should
> suggest based on few other fields like 'name','manu' etc. and due to
> fuzzylookupfactory it's give suggestion based on starting characters, not
> if users type any middle character of phrase eg. i tried with searching by
> 'stuff2' of 'electronics and stuff2' cat .
>
> curl 'http://localhost:8983/solr/techproducts/suggest?wt=json&ind
> ent=true&suggest.dictionary=mySuggester&suggest=true&suggest.q=stuff'
> {
> "responseHeader":{
> "status":0,
> "QTime":2},
> "suggest":{"mySuggester":{
> "stuff":{
> "numFound":0,
> "suggestions":[]}}}}
>
> On Fri, Apr 15, 2016 at 11:18 PM, Tejas Bhanushali <
> contact.tejasbhanushali@gmail.com> wrote:
>
>> Hi Team,
>>
>> Im getting the duplicate result when i do auto suggestion based on
>> multiple filed by using copy filed . i have below table configuration .
>>
>> Segment -- have multiple category -- have multiple sub category -- have
>> multiple products.
>>
>> suggestion are given based on
>> segment name, category name, sub category name and product name.
>>
>> below is output .
>>
>> ---------------
>>
>> {
>> "responseHeader": {
>> "status": 0,
>> "QTime": 1375
>> },
>> "command": "build",
>> "suggest": {
>> "mySuggester": {
>> "Fruit": {
>> "numFound": 10,
>> "suggestions": [
>> {
>> "term": "<b>Fruit</b>s & Vegetables",
>> "weight": 1000,
>> "payload": ""
>> },
>> {
>> "term": "<b>Fruit</b>s & Vegetables",
>> "weight": 1000,
>> "payload": ""
>> },
>> {
>> "term": "<b>Fruit</b>s & Vegetables",
>> "weight": 980,
>> "payload": ""
>> },
>> {
>> "term": "<b>Fruit</b>s & Vegetables",
>> "weight": 980,
>> "payload": ""
>> },
>> {
>> "term": "<b>Fruit</b>s & Vegetables",
>> "weight": 800,
>> "payload": ""
>> },
>> {
>> "term": "<b>Fruit</b>s & Vegetables",
>> "weight": 588,
>> "payload": ""
>> },
>> {
>> "term": "Cut <b>Fruit</b>s",
>> "weight": 456,
>> "payload": ""
>> },
>> {
>> "term": "<b>Fruit</b>s",
>> "weight": 456,
>> "payload": ""
>> },
>> {
>> "term": "<b>Fruit</b>s & Vegetables",
>> "weight": 456,
>> "payload": ""
>> },
>> {
>> "term": "<b>Fruit</b>s",
>> "weight": 456,
>> "payload": ""
>> }
>> ]
>> }
>> }
>> }
>> }
>>
>> --
>> Thanks & Regards,
>>
>> Tejas Bhanushali
>>
>
>
>
> --
> Thanks & Regards,
>
> Tejas Bhanushali
>
--
Thanks & Regards,
Tejas Bhanushali
On Mon, Apr 18, 2016 at 11:12 PM, Tejas Bhanushali <
contact.tejasbhanushali@gmail.com> wrote:
> HI Team,
>
> I tried to run the same example as suggested by Chris Hostetter and i get
> to know it's working fine for single field, but my requirement is it should
> suggest based on multiple fields .i.e not only on "cat" field but it should
> suggest based on few other fields like 'name','manu' etc. and due to
> fuzzylookupfactory it's give suggestion based on starting characters, not
> if users type any middle character of phrase eg. i tried with searching by
> 'stuff2' of 'electronics and stuff2' cat .
>
> curl 'http://localhost:8983/solr/techproducts/suggest?wt=json&ind
> ent=true&suggest.dictionary=mySuggester&suggest=true&suggest.q=stuff'
> {
> "responseHeader":{
> "status":0,
> "QTime":2},
> "suggest":{"mySuggester":{
> "stuff":{
> "numFound":0,
> "suggestions":[]}}}}
>
> On Fri, Apr 15, 2016 at 11:18 PM, Tejas Bhanushali <
> contact.tejasbhanushali@gmail.com> wrote:
>
>> Hi Team,
>>
>> Im getting the duplicate result when i do auto suggestion based on
>> multiple filed by using copy filed . i have below table configuration .
>>
>> Segment -- have multiple category -- have multiple sub category -- have
>> multiple products.
>>
>> suggestion are given based on
>> segment name, category name, sub category name and product name.
>>
>> below is output .
>>
>> ---------------
>>
>> {
>> "responseHeader": {
>> "status": 0,
>> "QTime": 1375
>> },
>> "command": "build",
>> "suggest": {
>> "mySuggester": {
>> "Fruit": {
>> "numFound": 10,
>> "suggestions": [
>> {
>> "term": "<b>Fruit</b>s & Vegetables",
>> "weight": 1000,
>> "payload": ""
>> },
>> {
>> "term": "<b>Fruit</b>s & Vegetables",
>> "weight": 1000,
>> "payload": ""
>> },
>> {
>> "term": "<b>Fruit</b>s & Vegetables",
>> "weight": 980,
>> "payload": ""
>> },
>> {
>> "term": "<b>Fruit</b>s & Vegetables",
>> "weight": 980,
>> "payload": ""
>> },
>> {
>> "term": "<b>Fruit</b>s & Vegetables",
>> "weight": 800,
>> "payload": ""
>> },
>> {
>> "term": "<b>Fruit</b>s & Vegetables",
>> "weight": 588,
>> "payload": ""
>> },
>> {
>> "term": "Cut <b>Fruit</b>s",
>> "weight": 456,
>> "payload": ""
>> },
>> {
>> "term": "<b>Fruit</b>s",
>> "weight": 456,
>> "payload": ""
>> },
>> {
>> "term": "<b>Fruit</b>s & Vegetables",
>> "weight": 456,
>> "payload": ""
>> },
>> {
>> "term": "<b>Fruit</b>s",
>> "weight": 456,
>> "payload": ""
>> }
>> ]
>> }
>> }
>> }
>> }
>>
>> --
>> Thanks & Regards,
>>
>> Tejas Bhanushali
>>
>
>
>
> --
> Thanks & Regards,
>
> Tejas Bhanushali
>
--
Thanks & Regards,
Tejas Bhanushali
Re: Getting duplicate output while doing auto suggestion based on
multiple filed using copy filed in solr 5.5
Posted by Tejas Bhanushali <co...@gmail.com>.
HI Team,
I tried to run the same example as suggested by Chris Hostetter and i get
to know it's working fine for single field, but my requirement is it should
suggest based on multiple fields .i.e not only on "cat" field but it should
suggest based on few other fields like 'name','manu' etc. and due to
fuzzylookupfactory it's give suggestion based on starting characters, not
if users type any middle character of phrase eg. i tried with searching by
'stuff2' of 'electronics and stuff2' cat .
curl 'http://localhost:8983/solr/techproducts/suggest?wt=json&ind
ent=true&suggest.dictionary=mySuggester&suggest=true&suggest.q=stuff'
{
"responseHeader":{
"status":0,
"QTime":2},
"suggest":{"mySuggester":{
"stuff":{
"numFound":0,
"suggestions":[]}}}}
On Fri, Apr 15, 2016 at 11:18 PM, Tejas Bhanushali <
contact.tejasbhanushali@gmail.com> wrote:
> Hi Team,
>
> Im getting the duplicate result when i do auto suggestion based on
> multiple filed by using copy filed . i have below table configuration .
>
> Segment -- have multiple category -- have multiple sub category -- have
> multiple products.
>
> suggestion are given based on
> segment name, category name, sub category name and product name.
>
> below is output .
>
> ---------------
>
> {
> "responseHeader": {
> "status": 0,
> "QTime": 1375
> },
> "command": "build",
> "suggest": {
> "mySuggester": {
> "Fruit": {
> "numFound": 10,
> "suggestions": [
> {
> "term": "<b>Fruit</b>s & Vegetables",
> "weight": 1000,
> "payload": ""
> },
> {
> "term": "<b>Fruit</b>s & Vegetables",
> "weight": 1000,
> "payload": ""
> },
> {
> "term": "<b>Fruit</b>s & Vegetables",
> "weight": 980,
> "payload": ""
> },
> {
> "term": "<b>Fruit</b>s & Vegetables",
> "weight": 980,
> "payload": ""
> },
> {
> "term": "<b>Fruit</b>s & Vegetables",
> "weight": 800,
> "payload": ""
> },
> {
> "term": "<b>Fruit</b>s & Vegetables",
> "weight": 588,
> "payload": ""
> },
> {
> "term": "Cut <b>Fruit</b>s",
> "weight": 456,
> "payload": ""
> },
> {
> "term": "<b>Fruit</b>s",
> "weight": 456,
> "payload": ""
> },
> {
> "term": "<b>Fruit</b>s & Vegetables",
> "weight": 456,
> "payload": ""
> },
> {
> "term": "<b>Fruit</b>s",
> "weight": 456,
> "payload": ""
> }
> ]
> }
> }
> }
> }
>
> --
> Thanks & Regards,
>
> Tejas Bhanushali
>
--
Thanks & Regards,
Tejas Bhanushali
Re: Getting duplicate output while doing auto suggestion based on
multiple filed using copy filed in solr 5.5
Posted by Erick Erickson <er...@gmail.com>.
You haven't responded to Chris's request for clarification.
Best,
Erick
On Fri, Apr 15, 2016 at 11:43 PM, Tejas Bhanushali
<co...@gmail.com> wrote:
> any update on it?
>
> On Fri, Apr 15, 2016 at 11:18 PM, Tejas Bhanushali <
> contact.tejasbhanushali@gmail.com> wrote:
>
>> Hi Team,
>>
>> Im getting the duplicate result when i do auto suggestion based on
>> multiple filed by using copy filed . i have below table configuration .
>>
>> Segment -- have multiple category -- have multiple sub category -- have
>> multiple products.
>>
>> suggestion are given based on
>> segment name, category name, sub category name and product name.
>>
>> below is output .
>>
>> ---------------
>>
>> {
>> "responseHeader": {
>> "status": 0,
>> "QTime": 1375
>> },
>> "command": "build",
>> "suggest": {
>> "mySuggester": {
>> "Fruit": {
>> "numFound": 10,
>> "suggestions": [
>> {
>> "term": "<b>Fruit</b>s & Vegetables",
>> "weight": 1000,
>> "payload": ""
>> },
>> {
>> "term": "<b>Fruit</b>s & Vegetables",
>> "weight": 1000,
>> "payload": ""
>> },
>> {
>> "term": "<b>Fruit</b>s & Vegetables",
>> "weight": 980,
>> "payload": ""
>> },
>> {
>> "term": "<b>Fruit</b>s & Vegetables",
>> "weight": 980,
>> "payload": ""
>> },
>> {
>> "term": "<b>Fruit</b>s & Vegetables",
>> "weight": 800,
>> "payload": ""
>> },
>> {
>> "term": "<b>Fruit</b>s & Vegetables",
>> "weight": 588,
>> "payload": ""
>> },
>> {
>> "term": "Cut <b>Fruit</b>s",
>> "weight": 456,
>> "payload": ""
>> },
>> {
>> "term": "<b>Fruit</b>s",
>> "weight": 456,
>> "payload": ""
>> },
>> {
>> "term": "<b>Fruit</b>s & Vegetables",
>> "weight": 456,
>> "payload": ""
>> },
>> {
>> "term": "<b>Fruit</b>s",
>> "weight": 456,
>> "payload": ""
>> }
>> ]
>> }
>> }
>> }
>> }
>>
>> --
>> Thanks & Regards,
>>
>> Tejas Bhanushali
>>
>
>
>
> --
> Thanks & Regards,
>
> Tejas Bhanushali
Re: Getting duplicate output while doing auto suggestion based on
multiple filed using copy filed in solr 5.5
Posted by Tejas Bhanushali <co...@gmail.com>.
any update on it?
On Fri, Apr 15, 2016 at 11:18 PM, Tejas Bhanushali <
contact.tejasbhanushali@gmail.com> wrote:
> Hi Team,
>
> Im getting the duplicate result when i do auto suggestion based on
> multiple filed by using copy filed . i have below table configuration .
>
> Segment -- have multiple category -- have multiple sub category -- have
> multiple products.
>
> suggestion are given based on
> segment name, category name, sub category name and product name.
>
> below is output .
>
> ---------------
>
> {
> "responseHeader": {
> "status": 0,
> "QTime": 1375
> },
> "command": "build",
> "suggest": {
> "mySuggester": {
> "Fruit": {
> "numFound": 10,
> "suggestions": [
> {
> "term": "<b>Fruit</b>s & Vegetables",
> "weight": 1000,
> "payload": ""
> },
> {
> "term": "<b>Fruit</b>s & Vegetables",
> "weight": 1000,
> "payload": ""
> },
> {
> "term": "<b>Fruit</b>s & Vegetables",
> "weight": 980,
> "payload": ""
> },
> {
> "term": "<b>Fruit</b>s & Vegetables",
> "weight": 980,
> "payload": ""
> },
> {
> "term": "<b>Fruit</b>s & Vegetables",
> "weight": 800,
> "payload": ""
> },
> {
> "term": "<b>Fruit</b>s & Vegetables",
> "weight": 588,
> "payload": ""
> },
> {
> "term": "Cut <b>Fruit</b>s",
> "weight": 456,
> "payload": ""
> },
> {
> "term": "<b>Fruit</b>s",
> "weight": 456,
> "payload": ""
> },
> {
> "term": "<b>Fruit</b>s & Vegetables",
> "weight": 456,
> "payload": ""
> },
> {
> "term": "<b>Fruit</b>s",
> "weight": 456,
> "payload": ""
> }
> ]
> }
> }
> }
> }
>
> --
> Thanks & Regards,
>
> Tejas Bhanushali
>
--
Thanks & Regards,
Tejas Bhanushali
Re: Getting duplicate output while doing auto suggestion based on
multiple filed using copy filed in solr 5.5
Posted by Chris Hostetter <ho...@fucit.org>.
I can't explain the results you are seeing, but you also didn't provide us
with your schema.xml (ie; how are "text" and "text_auto" defined?) or
enough details ot try and reproduce on a small scale (ie: what does the
source data look like in the documents where these suggestion values
are coming from.
If i start up the "bin/solr -e techproducts" example, which is also
configured to use DocumentDictionaryFactory, I don't see any duplicate
suggestions...
curl 'http://localhost:8983/solr/techproducts/suggest?suggest.dictionary=mySuggester&suggest=true&suggest.build=true&wt=json'
{"responseHeader":{"status":0,"QTime":13},"command":"build"}
curl 'http://localhost:8983/solr/techproducts/suggest?wt=json&indent=true&suggest.dictionary=mySuggester&suggest=true&suggest.q=elec'
{
"responseHeader":{
"status":0,
"QTime":1},
"suggest":{"mySuggester":{
"elec":{
"numFound":3,
"suggestions":[{
"term":"electronics and computer1",
"weight":2199,
"payload":""},
{
"term":"electronics",
"weight":649,
"payload":""},
{
"term":"electronics and stuff2",
"weight":279,
"payload":""}]}}}}
...can you provide us with some precises (and ideally minimal) steps to
reproduce the problem you are describing?
For Example...
1) "Add XYZ to the 5.5 sample_techproducts_configs solrconfig.xml"
2) "Add ABC to the 5.5 sample_techproducts_configs managed-schema"
3) run this curl command to index a few sample documents...
4) run this curl command to see some suggest results that have duplicates
in them based on the sample data from step #3
?
-Hoss
http://www.lucidworks.com/
Re: Getting duplicate output while doing auto suggestion based on
multiple filed using copy filed in solr 5.5
Posted by Tejas Bhanushali <co...@gmail.com>.
For more info PFA config file .
URL
http://localhost:8983/solr/products/suggest?suggest=true&suggest.build=true&wt=json&suggest.q=Fruit&echoParams=none
On Fri, Apr 15, 2016 at 11:18 PM, Tejas Bhanushali <
contact.tejasbhanushali@gmail.com> wrote:
> Hi Team,
>
> Im getting the duplicate result when i do auto suggestion based on
> multiple filed by using copy filed . i have below table configuration .
>
> Segment -- have multiple category -- have multiple sub category -- have
> multiple products.
>
> suggestion are given based on
> segment name, category name, sub category name and product name.
>
> below is output .
>
> ---------------
>
> {
> "responseHeader": {
> "status": 0,
> "QTime": 1375
> },
> "command": "build",
> "suggest": {
> "mySuggester": {
> "Fruit": {
> "numFound": 10,
> "suggestions": [
> {
> "term": "<b>Fruit</b>s & Vegetables",
> "weight": 1000,
> "payload": ""
> },
> {
> "term": "<b>Fruit</b>s & Vegetables",
> "weight": 1000,
> "payload": ""
> },
> {
> "term": "<b>Fruit</b>s & Vegetables",
> "weight": 980,
> "payload": ""
> },
> {
> "term": "<b>Fruit</b>s & Vegetables",
> "weight": 980,
> "payload": ""
> },
> {
> "term": "<b>Fruit</b>s & Vegetables",
> "weight": 800,
> "payload": ""
> },
> {
> "term": "<b>Fruit</b>s & Vegetables",
> "weight": 588,
> "payload": ""
> },
> {
> "term": "Cut <b>Fruit</b>s",
> "weight": 456,
> "payload": ""
> },
> {
> "term": "<b>Fruit</b>s",
> "weight": 456,
> "payload": ""
> },
> {
> "term": "<b>Fruit</b>s & Vegetables",
> "weight": 456,
> "payload": ""
> },
> {
> "term": "<b>Fruit</b>s",
> "weight": 456,
> "payload": ""
> }
> ]
> }
> }
> }
> }
>
> --
> Thanks & Regards,
>
> Tejas Bhanushali
>
--
Thanks & Regards,
Tejas Bhanushali