You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by cnyee <ye...@gmail.com> on 2011/12/14 10:50:01 UTC
Possible to adjust FieldNorm?
Hi,
Is it possible to adjust FieldNorm? I have a scenario where the search is
not producing the desired result because of fieldNorm:
Search terms: coaching leadership
Record 1: name="Ask the Coach", desc="...",...
Record 2: name="Coaching as a Leadership Development Tool Part 1",
desc="...",...
Record 1 was scored higher than record 2, despite record 2 has two matches.
The scoring is given below:
Record 1:
1.2878088 = (MATCH) weight(name_en:coach in 6430), product of:
0.20103075 = queryWeight(name_en:coach), product of:
6.406029 = idf(docFreq=160, maxDocs=35862)
0.03138149 = queryNorm
6.406029 = (MATCH) fieldWeight(name_en:coach in 6430), product of:
1.0 = tf(termFreq(name_en:coach)=1)
6.406029 = idf(docFreq=160, maxDocs=35862)
1.0 = fieldNorm(field=name_en, doc=6430)
Record 2:
0.56341636 = (MATCH) weight(name_en:coach in 4744), product of:
0.20103075 = queryWeight(name_en:coach), product of:
6.406029 = idf(docFreq=160, maxDocs=35862)
0.03138149 = queryNorm
2.8026378 = (MATCH) fieldWeight(name_en:coach in 4744), product of:
1.0 = tf(termFreq(crs_name_en:coach)=1)
6.406029 = idf(docFreq=160, maxDocs=35862)
0.4375 = fieldNorm(field=name_en, doc=4744)
Many thanks in advance.
Chut
--
View this message in context: http://lucene.472066.n3.nabble.com/Possible-to-adjust-FieldNorm-tp3584998p3584998.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Possible to adjust FieldNorm?
Posted by cnyee <ye...@gmail.com>.
Sorry, I did not give the full output in the first post.
For what it looks, the fieldNorm is saying that:
1 match out of 3 words in record 1 is more significant than 2 matches out of
8 words in record 2.
That would be true for simple arithmetic, but unsatisfactory in human
'meaning'.
Here are the full explanation. Record 2 has some boosting as well.
Record 1:
1.5843434 = (MATCH) sum of:
1.5372416 = (MATCH) sum of:
1.2878088 = (MATCH) max plus 0.1 times others of:
1.2878088 = (MATCH) weight(crs_name_en:coach in 6430), product of:
0.20103075 = queryWeight(crs_name_en:coach), product of:
6.406029 = idf(docFreq=160, maxDocs=35862)
0.03138149 = queryNorm
6.406029 = (MATCH) fieldWeight(crs_name_en:coach in 6430), product
of:
1.0 = tf(termFreq(crs_name_en:coach)=1)
6.406029 = idf(docFreq=160, maxDocs=35862)
1.0 = fieldNorm(field=crs_name_en, doc=6430)
0.2494328 = (MATCH) max plus 0.1 times others of:
0.2494328 = (MATCH) weight(crs_desc_en:leadership in 6430), product
of:
0.15826634 = queryWeight(crs_desc_en:leadership), product of:
5.043302 = idf(docFreq=628, maxDocs=35862)
0.03138149 = queryNorm
1.5760319 = (MATCH) fieldWeight(crs_desc_en:leadership in 6430),
product of:
1.0 = tf(termFreq(crs_desc_en:leadership)=1)
5.043302 = idf(docFreq=628, maxDocs=35862)
0.3125 = fieldNorm(field=crs_desc_en, doc=6430)
0.04710189 = (MATCH) product of:
0.09420378 = (MATCH) sum of:
0.09420378 = (MATCH) product of:
0.3768151 = (MATCH) sum of:
0.3768151 = (MATCH) weight(published_year:2008 in 6430), product
of:
0.10874291 = queryWeight(published_year:2008), product of:
3.4651926 = idf(docFreq=3047, maxDocs=35862)
0.03138149 = queryNorm
3.4651926 = (MATCH) fieldWeight(published_year:2008 in 6430),
product of:
1.0 = tf(termFreq(published_year:2008)=1)
3.4651926 = idf(docFreq=3047, maxDocs=35862)
1.0 = fieldNorm(field=published_year, doc=6430)
0.25 = coord(1/4)
0.5 = coord(1/2)
0.0 = (MATCH) FunctionQuery(int(crs_stars)), product of:
0.0 = int(crs_stars)=0
2.5 = boost
0.03138149 = queryNorm
Record 2:
1.5590522 = (MATCH) sum of:
1.0096307 = (MATCH) sum of:
0.6206793 = (MATCH) max plus 0.1 times others of:
0.56341636 = (MATCH) weight(crs_name_en:coach in 4744), product of:
0.20103075 = queryWeight(crs_name_en:coach), product of:
6.406029 = idf(docFreq=160, maxDocs=35862)
0.03138149 = queryNorm
2.8026378 = (MATCH) fieldWeight(crs_name_en:coach in 4744), product
of:
1.0 = tf(termFreq(crs_name_en:coach)=1)
6.406029 = idf(docFreq=160, maxDocs=35862)
0.4375 = fieldNorm(field=crs_name_en, doc=4744)
0.11664742 = (MATCH) weight(meta_en:coach in 4744), product of:
0.11443973 = queryWeight(meta_en:coach), product of:
3.646727 = idf(docFreq=2541, maxDocs=35862)
0.03138149 = queryNorm
1.0192913 = (MATCH) fieldWeight(meta_en:coach in 4744), product of:
2.236068 = tf(termFreq(meta_en:coach)=5)
3.646727 = idf(docFreq=2541, maxDocs=35862)
0.125 = fieldNorm(field=meta_en, doc=4744)
0.4559821 = (MATCH) weight(crs_desc_en:coach in 4744), product of:
0.19534174 = queryWeight(crs_desc_en:coach), product of:
6.2247434 = idf(docFreq=192, maxDocs=35862)
0.03138149 = queryNorm
2.3342788 = (MATCH) fieldWeight(crs_desc_en:coach in 4744), product
of:
2.0 = tf(termFreq(crs_desc_en:coach)=4)
6.2247434 = idf(docFreq=192, maxDocs=35862)
0.1875 = fieldNorm(field=crs_desc_en, doc=4744)
0.3889513 = (MATCH) max plus 0.1 times others of:
0.36372444 = (MATCH) weight(crs_name_en:leadership in 4744), product
of:
0.16152287 = queryWeight(crs_name_en:leadership), product of:
5.147074 = idf(docFreq=566, maxDocs=35862)
0.03138149 = queryNorm
2.251845 = (MATCH) fieldWeight(crs_name_en:leadership in 4744),
product of:
1.0 = tf(termFreq(crs_name_en:leadership)=1)
5.147074 = idf(docFreq=566, maxDocs=35862)
0.4375 = fieldNorm(field=crs_name_en, doc=4744)
0.04061773 = (MATCH) weight(meta_en:leadership in 4744), product of:
0.076728955 = queryWeight(meta_en:leadership), product of:
2.4450386 = idf(docFreq=8453, maxDocs=35862)
0.03138149 = queryNorm
0.5293664 = (MATCH) fieldWeight(meta_en:leadership in 4744), product
of:
1.7320508 = tf(termFreq(meta_en:leadership)=3)
2.4450386 = idf(docFreq=8453, maxDocs=35862)
0.125 = fieldNorm(field=meta_en, doc=4744)
0.21165074 = (MATCH) weight(crs_desc_en:leadership in 4744), product
of:
0.15826634 = queryWeight(crs_desc_en:leadership), product of:
5.043302 = idf(docFreq=628, maxDocs=35862)
0.03138149 = queryNorm
1.3373075 = (MATCH) fieldWeight(crs_desc_en:leadership in 4744),
product of:
1.4142135 = tf(termFreq(crs_desc_en:leadership)=2)
5.043302 = idf(docFreq=628, maxDocs=35862)
0.1875 = fieldNorm(field=crs_desc_en, doc=4744)
0.54942155 = (MATCH) sum of:
0.020222418 = (MATCH) weight(domain_ids:69^0.02 in 4744), product of:
0.0035626104 = queryWeight(domain_ids:69^0.02), product of:
0.02 = boost
5.6762924 = idf(docFreq=333, maxDocs=35862)
0.03138149 = queryNorm
5.6762924 = (MATCH) fieldWeight(domain_ids:69 in 4744), product of:
1.0 = tf(termFreq(domain_ids:69)=1)
5.6762924 = idf(docFreq=333, maxDocs=35862)
1.0 = fieldNorm(field=domain_ids, doc=4744)
0.5291991 = (MATCH) product of:
2.1167965 = (MATCH) sum of:
2.1167965 = (MATCH) weight(published_year:2011^7.0 in 4744), product
of:
0.68190736 = queryWeight(published_year:2011^7.0), product of:
7.0 = boost
3.104229 = idf(docFreq=4372, maxDocs=35862)
0.03138149 = queryNorm
3.104229 = (MATCH) fieldWeight(published_year:2011 in 4744),
product of:
1.0 = tf(termFreq(published_year:2011)=1)
3.104229 = idf(docFreq=4372, maxDocs=35862)
1.0 = fieldNorm(field=published_year, doc=4744)
0.25 = coord(1/4)
0.0 = (MATCH) FunctionQuery(int(crs_stars)), product of:
0.0 = int(crs_stars)=0
2.5 = boost
0.03138149 = queryNorm
--
View this message in context: http://lucene.472066.n3.nabble.com/Possible-to-adjust-FieldNorm-tp3584998p3587145.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Possible to adjust FieldNorm?
Posted by Chris Hostetter <ho...@fucit.org>.
: From what I can see, the problem there is not with the field norm, but with
: the fact that "leadership" is not matching the second document for some
: reason. Is it possible that you are having some kind of analysis problem?
Agreed ... if those are your full score explanations for those two
documents then something is not right with how the second document is
matcihng your query.
what exactly does your request look like? what exactly does the
requestHandler configuration look like? what is the final parsed query
according to the debug information? what does the fieldtype for "name_en"
look like? what does analysis.jsp say about how those name_en field
values are analized at index time? how is the word "leadership" analyzed
at query time?
In general, if you want to disable norms you can set "omitNorm" on the
field -- or you can customize the similariy to change the lengthNorm
function.
: > Search terms: coaching leadership
: > Record 1: name="Ask the Coach", desc="...",...
: > Record 2: name="Coaching as a Leadership Development Tool Part 1",
: > desc="...",...
: >
: > Record 1 was scored higher than record 2, despite record 2 has two matches.
: > The scoring is given below:
: >
: > Record 1:
: > 1.2878088 = (MATCH) weight(name_en:coach in 6430), product of:
: > 0.20103075 = queryWeight(name_en:coach), product of:
: > 6.406029 = idf(docFreq=160, maxDocs=35862)
: > 0.03138149 = queryNorm
: > 6.406029 = (MATCH) fieldWeight(name_en:coach in 6430), product of:
: > 1.0 = tf(termFreq(name_en:coach)=1)
: > 6.406029 = idf(docFreq=160, maxDocs=35862)
: > 1.0 = fieldNorm(field=name_en, doc=6430)
: >
: > Record 2:
: > 0.56341636 = (MATCH) weight(name_en:coach in 4744), product of:
: > 0.20103075 = queryWeight(name_en:coach), product of:
: > 6.406029 = idf(docFreq=160, maxDocs=35862)
: > 0.03138149 = queryNorm
: > 2.8026378 = (MATCH) fieldWeight(name_en:coach in 4744), product of:
: > 1.0 = tf(termFreq(crs_name_en:coach)=1)
: > 6.406029 = idf(docFreq=160, maxDocs=35862)
: > 0.4375 = fieldNorm(field=name_en, doc=4744)
-Hoss
Re: Possible to adjust FieldNorm?
Posted by Tomás Fernández Löbbe <to...@gmail.com>.
>From what I can see, the problem there is not with the field norm, but with
the fact that "leadership" is not matching the second document for some
reason. Is it possible that you are having some kind of analysis problem?
On Wed, Dec 14, 2011 at 6:50 AM, cnyee <ye...@gmail.com> wrote:
> Hi,
>
> Is it possible to adjust FieldNorm? I have a scenario where the search is
> not producing the desired result because of fieldNorm:
>
> Search terms: coaching leadership
> Record 1: name="Ask the Coach", desc="...",...
> Record 2: name="Coaching as a Leadership Development Tool Part 1",
> desc="...",...
>
> Record 1 was scored higher than record 2, despite record 2 has two matches.
> The scoring is given below:
>
> Record 1:
> 1.2878088 = (MATCH) weight(name_en:coach in 6430), product of:
> 0.20103075 = queryWeight(name_en:coach), product of:
> 6.406029 = idf(docFreq=160, maxDocs=35862)
> 0.03138149 = queryNorm
> 6.406029 = (MATCH) fieldWeight(name_en:coach in 6430), product of:
> 1.0 = tf(termFreq(name_en:coach)=1)
> 6.406029 = idf(docFreq=160, maxDocs=35862)
> 1.0 = fieldNorm(field=name_en, doc=6430)
>
> Record 2:
> 0.56341636 = (MATCH) weight(name_en:coach in 4744), product of:
> 0.20103075 = queryWeight(name_en:coach), product of:
> 6.406029 = idf(docFreq=160, maxDocs=35862)
> 0.03138149 = queryNorm
> 2.8026378 = (MATCH) fieldWeight(name_en:coach in 4744), product of:
> 1.0 = tf(termFreq(crs_name_en:coach)=1)
> 6.406029 = idf(docFreq=160, maxDocs=35862)
> 0.4375 = fieldNorm(field=name_en, doc=4744)
>
> Many thanks in advance.
>
> Chut
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Possible-to-adjust-FieldNorm-tp3584998p3584998.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>