Appropriate steps for mapred

[Michael Taggart <mi...@webco.tv>]

I have followed the tutorial at media-style.com and actually have a
mapred installation of nutch working. Thanks Stefan :)
My question now is the correct steps to continuously fetch and index. I
have read some people talking about mergesegs and updatedb however
Stefan's tutorial doesn't list these as steps. If you want to
continually fetch more and more levels from your crawldb and
appropriately update your index what is the correct method for doing so?
Currently I am doing this:
generate
fetch
invertlinks
index

Only problem I am having is that I seem to not be able to get any pages
past the index pages on the root domains I injected. I feel like I am
missing some important steps. Any input is appreciated.
Mike

Re: Appropriate steps for mapred

[Stefan Groschupf <sg...@media-style.com>]

> Stefan's tutorial doesn't list these as steps.
I will add these steps hopefully until this year.
> If you want to
> continually fetch more and more levels from your crawldb and
> appropriately update your index what is the correct method for  
> doing so?
> Currently I am doing this:
> generate
> fetch
> invertlinks
> index

Looks like you missed to update the crawldb after fetching, but in  
general that is the way to go.
You can run this cycle 100000 times or more :). I suggest have big  
enough segments size and later merging some indexes together.
Just play around and try it out.

The size of segments and how many segment indexes you should merge  
very much depends on your hardware.
Also note that searching from a index stored on ndfs is slow, but  
there will be a solution for that until next weeks or so.

HTH
Stefan