Re: OPIC score calculation issues

[Andrzej Bialecki <ab...@getopt.org>]

(Better late than never... I forgot I didn't yet respond to your posting).

Doug Cutting wrote:
> I think all that you're saying is that we should not run two CrawlDB 
> updates at once, right?  But there are lots of reasons we cannot do 
> that besides the OPIC calculation.

When we used WebDB it was possible to overlap generate / fetch / update 
cycles, because we would "lock" pages selected by FetchListTool for a 
period of time.

Now we don't do this. The advantage is that we don't have to rewrite 
CrawlDB. But operations on CrawlDB are considerably faster than on 
WebDB, perhaps we should consider going back to this method?

>
>> Also, the "cash value" of those outlinks that point to URLs not in 
>> the current fetchlist will be dropped, because they won't be 
>> collected anywhere.
>
> No, every cash value is used.  The input to a crawl db update includes a
> CrawlDatum for every known url, including those just linked to.  If 
> the only CrawlDatum for a url is a new outlink from a page crawled, 
> then the  score for the page is 1.0 + the score of that outlink.

Of course, you are right, I missed this.


>> And a final note: CrawlDB.update() uses the initial score value 
>> recorded in the segment, and NOT the value that is actually found in 
>> CrawlDB at the time of the update. This means that if there was 
>> another update in the meantime, your new score in CrawlDB will be 
>> overwritten with the score based on an older initial value. This is 
>> counter-intuitive - I think CrawlDB.update() should always use the 
>> latest score value found in the current CrawlDB. I.e. in 
>> CrawlDBReducer instead of doing:
>>
>>      result.setScore(result.getScore() + scoreIncrement);
>>
>> we should do:
>>
>>      result.setScore(old.getScore() + scoreIncrement);
>
> The change is not quite that simple, since 'old' is sometimes null.  
> So perhaps we need to add an 'score' variable that is set to old.score 
> when old!=null and to 1.0 otherwise (for newly linked pages).
>
> The reason I didn't do it that way was to permit the Fetcher to modify 
> scores, since I was thinking of the Fetcher as the actor whose actions 
> are being processed here, and of the CrawlDb as the passive thing 
> acted on.  But indeed, if you have another process that's updating a 
> CrawlDb while a Fetcher is running, this may not be the case.  So if 
> we want to switch things so that the Fetcher is not permitted to 
> adjust scores, then this seems like a reasonable change.

I would vote for implementing this change. The reason is that the active 
actor that computes new scores is CrawlDb.update(). Fetcher may provide 
additional information to affect the score, but IMHO the logic to 
calculate new scores should be concentrated in the update() method.

-- 
Best regards,
Andrzej Bialecki     <><
 ___. ___ ___ ___ _ _   __________________________________
[__ || __|__/|__||\/|  Information Retrieval, Semantic Web
___|||__||  \|  ||  |  Embedded Unix, System Integration
http://www.sigram.com  Contact: info at sigram dot com

Re: OPIC score calculation issues

[Doug Cutting <cu...@apache.org>]

Andrzej Bialecki wrote:
> When we used WebDB it was possible to overlap generate / fetch / update 
> cycles, because we would "lock" pages selected by FetchListTool for a 
> period of time.
> 
> Now we don't do this. The advantage is that we don't have to rewrite 
> CrawlDB. But operations on CrawlDB are considerably faster than on 
> WebDB, perhaps we should consider going back to this method?

Yes, this would be a good addition.

Ideally we should change Crawl.java to overlap these too.  When -topN is 
specified and substantially smaller than the total size of the crawldb, 
then we can generate, start a fetch job, then generate again.  As each 
fetch completes, we can start the next, then run an update and generate 
based on the just-completed fetch, so that we're constantly fetching.

This could be implemented by: (a) adding a status for generated crawl 
data; (b) adding a option to updatedb to include the generated output 
from some segments.  Then, in the above algorithm, the first time we'd 
update with only the generator output, but, after that, we can combine 
the updates with fetcher and generator output.  This way, in the course 
of a crawl, we only re-write the crawldb one additional time, rather 
than twice as many times.  Does this make sense?

>>> And a final note: CrawlDB.update() uses the initial score value 
>>> recorded in the segment, and NOT the value that is actually found in 
>>> CrawlDB at the time of the update. This means that if there was 
>>> another update in the meantime, your new score in CrawlDB will be 
>>> overwritten with the score based on an older initial value. This is 
>>> counter-intuitive - I think CrawlDB.update() should always use the 
>>> latest score value found in the current CrawlDB. I.e. in 
>>> CrawlDBReducer instead of doing:
>>>
>>>      result.setScore(result.getScore() + scoreIncrement);
>>>
>>> we should do:
>>>
>>>      result.setScore(old.getScore() + scoreIncrement);
>>
>>
>> The change is not quite that simple, since 'old' is sometimes null.  
>> So perhaps we need to add an 'score' variable that is set to old.score 
>> when old!=null and to 1.0 otherwise (for newly linked pages).
>>
>> The reason I didn't do it that way was to permit the Fetcher to modify 
>> scores, since I was thinking of the Fetcher as the actor whose actions 
>> are being processed here, and of the CrawlDb as the passive thing 
>> acted on.  But indeed, if you have another process that's updating a 
>> CrawlDb while a Fetcher is running, this may not be the case.  So if 
>> we want to switch things so that the Fetcher is not permitted to 
>> adjust scores, then this seems like a reasonable change.
> 
> I would vote for implementing this change. The reason is that the active 
> actor that computes new scores is CrawlDb.update(). Fetcher may provide 
> additional information to affect the score, but IMHO the logic to 
> calculate new scores should be concentrated in the update() method.

I agree: +1.  I was just trying to explain the existing logic.  I think 
this would provide a significant improvement, with little lost.

Doug