Question about injecting, generating fetch segment and refetching

[Juho Mäkinen <ju...@gmail.com>]

I managed to extract urld from segments which fetcher failed
to fetch for some reason. I'm now thinking what's the best
way to refetch those urls again? I was first thinking to
creating another db/segments pair, inject these urls into the new webdb,
fetching them and then merking the results back to the main database.
Is there a better way?

Another question is about injecting urls into the webdb. I first inject
some seed urls into the webdb and then starts fetching them, that's ok.
But the bin/nutch generate command creates a new segment,
with urls to fetch, from the webdb, right? What does the -topN parameter
exactly do? Does it get N urls from the web db which has the greates
rate/value/score, or does it simply get N urls from the webdb which
has been pushed onto the "top" (is there a "top" in the webdb?) of the webdb?

Last question is about refetching pages after a short while. I can construct
a list of urls which has been added into the site (direcly from the filesystem
using find -command), what's the best way to add them into the main database?
And how should I create a new segment for fetching with urls which has not
been indexed for N days?

Thanks in advantage. I'll promise to add the answers to the Wiki as good
as I can document them, because I bet that there's more users which
are thinking the same questiond :/

 - Juho Mäkinen

Re: Question about injecting, generating fetch segment and refetching

[Matthias Jaekle <ja...@eventax.de>]

> I have a question about -topN. Does -topN only work after having done a
> round of fetching? 
If you have never fetched some pages, and run updatedb, so, if you only 
injected some pages, all pages should be scored same. So using -topN 
does not make sense.

> What happens if I use -topN when generating a segment
> to fetch before I've fetched anything? Will it just select a random subset
> of N URLs, or will it use the first N URLs?
Please have a look at the code for the implementation.

Matthias

Re: Question about injecting, generating fetch segment and refetching

[sh...@cs.uoregon.edu]

Hello kind sirs,

>> Another question is about injecting urls into the webdb. I first inject
>> some seed urls into the webdb and then starts fetching them, that's ok.
>> But the bin/nutch generate command creates a new segment,
>> with urls to fetch, from the webdb, right? What does the -topN parameter
>> exactly do? Does it get N urls from the web db which has the greates
>> rate/value/score, or does it simply get N urls from the webdb which
>> has been pushed onto the "top" (is there a "top" in the webdb?) of the
>> webdb?
> Correct, -topN should use the best scored urls form db.

I have a question about -topN. Does -topN only work after having done a
round of fetching? What happens if I use -topN when generating a segment
to fetch before I've fetched anything? Will it just select a random subset
of N URLs, or will it use the first N URLs?

Many thanks,

-Shiwoong

Re: Question about injecting, generating fetch segment and refetching

[Matthias Jaekle <ja...@eventax.de>]

> I managed to extract urld from segments which fetcher failed
> to fetch for some reason. I'm now thinking what's the best
> way to refetch those urls again? I was first thinking to
> creating another db/segments pair, inject these urls into the new webdb,
> fetching them and then merking the results back to the main database.
> Is there a better way?
Updatedb with the segment where fetcher failed.
Then generate a new segments.
Stuff which was not fetched whould be resceduled after 7 days, I think.
If this time is not over, you could use -adddays option of the generate 
modul.


> Another question is about injecting urls into the webdb. I first inject
> some seed urls into the webdb and then starts fetching them, that's ok.
> But the bin/nutch generate command creates a new segment,
> with urls to fetch, from the webdb, right? What does the -topN parameter
> exactly do? Does it get N urls from the web db which has the greates
> rate/value/score, or does it simply get N urls from the webdb which
> has been pushed onto the "top" (is there a "top" in the webdb?) of the webdb?
Correct, -topN should use the best scored urls form db.



> Last question is about refetching pages after a short while. I can construct
> a list of urls which has been added into the site (direcly from the filesystem
> using find -command), what's the best way to add them into the main database?
If you would crawl the whole page, you would also have the anchor texts 
in your db.


> And how should I create a new segment for fetching with urls which has not
> been indexed for N days?
Have a look at the times speciefied in the config files and use the 
-adddays option from the generate modul.

Matthias
-- 
http://www.eventax.com - eventax GmbH
http://www.umkreisfinder.de - Die Suchmaschine für Lokales und Events