[Nutch Wiki] Update of "SimpleMapReduceTutorial" by EarlCahill

[Apache Wiki <wi...@apache.org>]

Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Nutch Wiki" for change notification.

The following page has been changed by EarlCahill:
http://wiki.apache.org/nutch/SimpleMapReduceTutorial

New page:
This is the simplest map reduce example I could come up with. Local filesystem, just getting one segment indexed. I am running Ubuntu, on an Athlon 3200+ using a cable modem connection.

== Designate Url ==

Need to get to the right place

{{{
cd nutch/branches/mapred
}}}

We need to make a directory that contains files, where each line of each file is a url. I choose http://lucene.apache.org/nutch/

{{{
mkdir urls
echo "http://lucene.apache.org/nutch/" > urls/urls
}}}

Also need to change the crawl filter to include this site

{{{
perl -pi -e 's|MY.DOMAIN.NAME|lucene.apache.org/nutch|' conf/crawl-urlfilter.txt
}}}

We walk through the following steps: crawl, generate, fetch, updatedb, invertlinks, index.

== Crawl ==

We want to run crawl on the urls directory from above.

{{{
./bin/nutch crawl urls
}}}

Took me about ten minutes. Output included

051004 003916 178 pages, 17 errors, 0.4 pages/s, 48 kb/s

The errors generally seemed to be timeouts.

The rest of the commands are a bit more dynamic, relying on timestamp and the like. Environment variables help out.

== Generate ==

Here we walk a segment dir from the crawl above.

{{{
CRAWLDB=`find crawl-2* -name crawldb`
SEGMENTS_DIR=`find crawl-2* -maxdepth 1 -name segments`
./bin/nutch generate $CRAWLDB $SEGMENTS_DIR
}}}

Took less than five seconds.

== Fetch ==

{{{
SEGMENT=`find crawl-2*/segments/2* -maxdepth 0 | tail -1`
./bin/nutch fetch $SEGMENT
}}}

Took about seven minutes, and output looked like

051004 004931 65 pages, 404 errors, 0.2 pages/s, 19 kb/s,

Again, many timeouts.

== UbdateDB ==

{{{
./bin/nutch updatedb $CRAWLDB $SEGMENT
}}}

Took less than five seconds.

== InvertLinks ==

{{{
LINKDB=`find crawl-2* -name linkdb -maxdepth 1`
SEGMENTS=`find crawl-2* -name segments -maxdepth 1`
./bin/nutch invertlinks $LINKDB $SEGMENTS
}}}

Took less than five seconds.

== Index ==

We need a place for our index, say myindex

{{{
mkdir myindex
}}}

Now, let's index.

{{{
./bin/nutch index myindex $LINKDB $SEGMENT
}}}

Took less than ten seconds.

== Test ==

The best test I have for the moment is

{{{
ls -alR myindex
}}}

If you see several files, it at least did something. Happy nutching!

Tutorial written by Earl Cahill, 2005.