You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by POIRIER David <DP...@cross-systems.com> on 2008/06/06 17:44:24 UTC
score calculation
Hello,
My index is, I guess, a little bit unorthodox: out of 75 000 entries, 55
000 comes from a single source and the last 20 000 from 15 other
sources. When executing a query though I would like to see all the
sources on the same starting line, and it doesn't seem to be the case:
the documents coming from the "big" source are always last in the
results lists.
I then checked how the score is calculated:
score(q,d) = coord(q,d) . queryNorm(q) . SUM( tf(t in d) . idf(t)^2
t.getBoost() . norm(t, d) )
where q is a query, t is a term and d a document
I won't go into all the details, for that check the following link:
http://hudson.zones.apache.org/hudson/job/Lucene-trunk/javadoc//org/apac
he/lucene/search/Similarity.html
<http://hudson.zones.apache.org/hudson/job/Lucene-trunk/javadoc/org/apac
he/lucene/search/Similarity.html> but using excellent Luke application,
written by Andrezej Bialecki, I understood that my problem is due to the
last element of the equation: norm(t, d)..
Using Luke I checked the detailed explanation of a document after
executing a simple query on the index, something like
content:"anExpression". The score was equal to tf( ) * idf() *
fieldNorm( ), and in there, fieldNorm = 0.0001, completely killing the
score. this fieldNorn() value is, I guess, equal to the t.getBoost( ) .
norm( ) we see in the main equation? When checking the details of the
document fields for all the documents in Luke, I could read that the
boost field was amazingly low at 0.00419... Why is that? What is this
boost value exactly? The document boost?
Note: Based on Lucene documentation, norm( ) encapsulate the document
boost, the field boost and the lengthNorm( ) value. This lengthNorm( )
value is computed when the document is added to the index in accordance
with the number of tokens of this field in this document. It should, in
theory, be of no impact here.
I went a little bit further, creating 3 indexes for 3 different sources
(one index per source). I got the following metrics in Luke:
Index size: 190
boost value for all documents: 0.08222
fieldNorm( ) value for all documents = 0.0024
Index size: 789
boost value for all documents: 0.067
fieldNorm( ) value for all documents = 0.0020
Index size: 6838
boost value for all documents: 0.012
fieldNorm( ) value for all documents = 0.0004
Index size: 50 000
boost value for all documents: 0.004193
fieldNorm( ) value for all documents = 0.0001
As we can see there is a clear correlation between the size of the index
and the boost value associated with. Take note that when looking at a
single source, the size and URL patterns of every referenced documents
are alike.
There is indeed something strange. If we agree that the fieldNorm( )
value is function of the boost value, then I have a problem with the
boost. The boost value displayed in Luke should be the same! Except if
the document boost or the field boost is, somehow, linked to the size of
the index... or the size of a segment? or the URL pattern?
Basically my question is: How is the boost value displayed in Luke as a
field for every indexed document calculated?
Thank you and good week-end,
David
RE: score calculation
Posted by POIRIER David <DP...@cross-systems.com>.
You are partially right:
the method
org.apache.nutch.scoring.opic.OPICScoringFilter.distributeScoreToOutlink
is
indeed influencing the document boost value depending on the number of
outlinks its "parent document" has (I think). This is pretty
straitforward; the two key lines of code:
score /= allCount;
score /= validCount;
But if what you want is to have a constant boost value of 1.0 for every
indexed document, the key method is
org.apache.nutch.scoring.opic.OPICScoringFilter.indexerScore. It returns
the document boost value.
Hope it helps!
David
-----Original Message-----
From: vanderkerkoff [mailto:mjdavies@glam.ac.uk]
Sent: lundi, 9. juin 2008 15:47
To: nutch-user@lucene.apache.org
Subject: RE: score calculation
Hi David
I notice that a developer has pointed something very similar to what
you're
describing in the code of OPICScoringFilter.java itself.
Sorry for being twp, but is this the bit I need to edit? I've never
done
any java at all before.
/** Get a float value from Fetcher.SCORE_KEY, divide it by the number of
outlinks and apply. */
public CrawlDatum distributeScoreToOutlinks(Text fromUrl, ParseData
parseData, Collection<Entry<Text, CrawlDatum>> targets, CrawlDatum
adjust,
int allCount) throws ScoringFilterException {
float score = scoreInjected;
String scoreString =
parseData.getContentMeta().get(Nutch.SCORE_KEY);
if (scoreString != null) {
try {
score = Float.parseFloat(scoreString);
} catch (Exception e) {
e.printStackTrace(LogUtil.getWarnStream(LOG));
}
}
int validCount = targets.size();
if (countFiltered) {
score /= allCount;
} else {
if (validCount == 0) {
// no outlinks to distribute score, so just return adjust
return adjust;
}
score /= validCount;
}
// internal and external score factor
float internalScore = score * internalScoreFactor;
float externalScore = score * externalScoreFactor;
for (Entry<Text, CrawlDatum> target : targets) {
try {
String toHost = new URL(target.getKey().toString()).getHost();
String fromHost = new URL(fromUrl.toString()).getHost();
if(toHost.equalsIgnoreCase(fromHost)){
target.getValue().setScore(internalScore);
} else {
target.getValue().setScore(externalScore);
}
} catch (MalformedURLException e) {
e.printStackTrace(LogUtil.getWarnStream(LOG));
target.getValue().setScore(externalScore);
}
}
// XXX (ab) no adjustment? I think this is contrary to the algorithm
descr.
// XXX in the paper, where page "loses" its score if it's
distributed to
// XXX linked pages...
return adjust;
}
--
View this message in context:
http://www.nabble.com/score-calculation-tp17695314p17733194.html
Sent from the Nutch - User mailing list archive at Nabble.com.
RE: score calculation
Posted by vanderkerkoff <mj...@glam.ac.uk>.
Hi David
I notice that a developer has pointed something very similar to what you're
describing in the code of OPICScoringFilter.java itself.
Sorry for being twp, but is this the bit I need to edit? I've never done
any java at all before.
/** Get a float value from Fetcher.SCORE_KEY, divide it by the number of
outlinks and apply. */
public CrawlDatum distributeScoreToOutlinks(Text fromUrl, ParseData
parseData, Collection<Entry<Text, CrawlDatum>> targets, CrawlDatum adjust,
int allCount) throws ScoringFilterException {
float score = scoreInjected;
String scoreString = parseData.getContentMeta().get(Nutch.SCORE_KEY);
if (scoreString != null) {
try {
score = Float.parseFloat(scoreString);
} catch (Exception e) {
e.printStackTrace(LogUtil.getWarnStream(LOG));
}
}
int validCount = targets.size();
if (countFiltered) {
score /= allCount;
} else {
if (validCount == 0) {
// no outlinks to distribute score, so just return adjust
return adjust;
}
score /= validCount;
}
// internal and external score factor
float internalScore = score * internalScoreFactor;
float externalScore = score * externalScoreFactor;
for (Entry<Text, CrawlDatum> target : targets) {
try {
String toHost = new URL(target.getKey().toString()).getHost();
String fromHost = new URL(fromUrl.toString()).getHost();
if(toHost.equalsIgnoreCase(fromHost)){
target.getValue().setScore(internalScore);
} else {
target.getValue().setScore(externalScore);
}
} catch (MalformedURLException e) {
e.printStackTrace(LogUtil.getWarnStream(LOG));
target.getValue().setScore(externalScore);
}
}
// XXX (ab) no adjustment? I think this is contrary to the algorithm
descr.
// XXX in the paper, where page "loses" its score if it's distributed to
// XXX linked pages...
return adjust;
}
--
View this message in context: http://www.nabble.com/score-calculation-tp17695314p17733194.html
Sent from the Nutch - User mailing list archive at Nabble.com.
RE: score calculation
Posted by POIRIER David <DP...@cross-systems.com>.
Matt,
There is one key thing that I have just understood: how Nutch calculates
the boost value found in the boost metadata field that you find with
every document inside your index: the scoring-opic plugin.
As you probably already know, Nutch is being developed using a
plugin-based architecture. By creating new plugins you can modify the
Nutch's behavior when crawling a source. Common modifications involve
parsing plugins and query plugins; more details on that are available on
the wiki.
But what I have found this morning is that the scoric-opic plugin is not
called when a search is executed on an index, as I though, but when the
fetched documents are being parsed. The score calculated by this plugin
is, I might be mistaking but I don't think so, the document boost value
and not the actual score linked to a document and based on a query. I
will call it the docScoreRelatedToAQuery.
As you now know, the docScoreRelatedToAQuery formula is:
docScoreRelatedToAQuery (q,d) = coord(q,d) . queryNorm(q) . SUM( tf(t in
d) . idf(t)^2 t.getBoost() . norm(t, d) )
Where
norm(t, d) = doc.getBoost() . lengthNorm(field) . f.getBoost()
If we take for granted that every generated field has a default boost
value of 1.0, the equation becomes:
norm(t, d) = doc.getBoost() . lengthNorm(field)
And this is where a problem arises for me. While the lengthNorm (field)
value was ok, the doc.getBoost() was returning astronomically low values
for certain indexes.
Looking at the source code of the scoring-opic plugin I understood that
this value was directly linked to the number of outlinks that the parent
document has. Basically, if your seed page is a generated page with
thousands of links toward child documents, those children will see there
doc.getBoost() value be extremely low and this will directly impact
their position inside a result set.
I modified the plugin, eliminating the unwanted division, and I now have
a doc.getBoost() value of 1.0 for every document. In my case, the norm
equation is now:
norm(t, d) = lengthNorm(field)
Note: the involved java class is
org.apache.nutch.scoring.opic.OPICScoringFilter found under the
${NUTCH_HOME}/src/plugins/scoring-opic directory
And things are good again and I will be able to enjoy tonight's Italy vs
Netherland game :-)
Hope this helps,
David
-----Original Message-----
From: vanderkerkoff [mailto:mjdavies@glam.ac.uk]
Sent: lundi, 9. juin 2008 12:09
To: nutch-user@lucene.apache.org
Subject: Re: score calculation
Hi David
I'm trying to understand the same thing, how the scoring works when you
have
more than one site being indexed.
I'll keep you informed if I work anything out, although I'm a bit behind
you
for certain.
I'm playing catch up :-)
Matt
--
View this message in context:
http://www.nabble.com/score-calculation-tp17695314p17729867.html
Sent from the Nutch - User mailing list archive at Nabble.com.
Re: score calculation
Posted by vanderkerkoff <mj...@glam.ac.uk>.
Hi David
I'm trying to understand the same thing, how the scoring works when you have
more than one site being indexed.
I'll keep you informed if I work anything out, although I'm a bit behind you
for certain.
I'm playing catch up :-)
Matt
--
View this message in context: http://www.nabble.com/score-calculation-tp17695314p17729867.html
Sent from the Nutch - User mailing list archive at Nabble.com.