You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Christopher Gross <co...@gmail.com> on 2011/12/19 19:41:27 UTC
problem with tutorial
I created a script based on:
http://wiki.apache.org/nutch/NutchTutorial#A3.2_Using_Individual_Commands_for_Whole-Web_Crawling
The script is similar to what happens in the old runbot.sh script, but
it isn't working for me. The part that does the "s1" barely gets
anything, but then the "s2" fails completely (it tries to make the
same segment?).
The plain "crawl" command seems to work, but how practical is it to
use that instead of making this script? Is it better to have all the
parts broken down like this, instead of passing more parts to the
"crawl" command and letting that handle things?
Any help would be greatly appreciated!
Script:
# runbot script to run the Nutch bot for crawling and re-crawling.
# Usage: bin/runbot [safe]
# If executed in 'safe' mode, it doesn't delete the temporary
# directories generated during crawl. This might be helpful for
# analysis and recovery in case a crawl fails.
#
# Author: Susam Pal
#Set up some local vars -- like NUTCH_HOME & CATALINA_HOME
. /common/setenv.sh
CRAWL_DIR=$NUTCH_HOME/crawl
SEGMENTS_DIR=$CRAWL_DIR/segments
CRAWL_DB=$CRAWL_DIR/crawldb
LINK_DB=$CRAWL_DIR/linkdb
depth=5
threads=5
adddays=5
topN=15 #Comment this statement if you don't want to set topN value
# Arguments for rm and mv
RMARGS="-rf"
MVARGS="--verbose"
# Parse arguments
if [ "$1" == "safe" ]
then
safe=yes
fi
if [ -z "$NUTCH_HOME" ]
then
NUTCH_HOME=/cdda/nutch
echo runbot: $0 could not find environment variable NUTCH_HOME
echo runbot: NUTCH_HOME=$NUTCH_HOME has been set by the script
else
echo runbot: $0 found environment variable NUTCH_HOME=$NUTCH_HOME
fi
if [ -z "$CATALINA_HOME" ]
then
CATALINA_HOME=/cdda/apache-tomcat
echo runbot: $0 could not find environment variable NUTCH_HOME
echo runbot: CATALINA_HOME=$CATALINA_HOME has been set by the script
else
echo runbot: $0 found environment variable CATALINA_HOME=$CATALINA_HOME
fi
if [ -n "$topN" ]
then
topN="-topN $topN"
else
topN=""
fi
echo "crawl.sh: STARTING"
echo $NUTCH_HOME/bin/nutch inject $CRAWL_DB $NUTCH_HOME/urls
echo ""
$NUTCH_HOME/bin/nutch inject $CRAWL_DB $NUTCH_HOME/urls
echo $NUTCH_HOME/bin/nutch generate $NUTCH_HOME/crawl/crawldb
$SEGMENTS_DIR -numFetchers 5
echo ""
$NUTCH_HOME/bin/nutch generate $CRAWL_DB $SEGMENTS_DIR -numFetchers 5
s1=`ls -rd $SEGMENTS_DIR/* | tail -1`
echo Segments: $s1
echo ""
echo $NUTCH_HOME/bin/nutch fetch $s1 -threads $threads
echo ""
$NUTCH_HOME/bin/nutch fetch $s1 -threads $threads
echo $NUTCH_HOME/bin/nutch parse $s1
echo ""
$NUTCH_HOME/bin/nutch parse $s1
echo $NUTCH_HOME/bin/nutch updatedb $CRAWL_DB $s1
echo ""
$NUTCH_HOME/bin/nutch updatedb $CRAWL_DB $s1
echo $NUTCH_HOME/bin/nutch generate crawldb $SEGMENTS_DIR -numFetchers 5
echo ""
$NUTCH_HOME/bin/nutch generate crawldb $SEGMENTS_DIR -numFetchers 5
s2=`ls -rd $SEGMENTS_DIR/* | tail -1`
echo Segments: $s2
echo ""
echo $NUTCH_HOME/bin/nutch fetch $s2 -threads $threads
echo ""
$NUTCH_HOME/bin/nutch fetch $s2 -threads $threads
echo $NUTCH_HOME/bin/nutch parse $s2
echo ""
$NUTCH_HOME/bin/nutch parse $s2
echo $NUTCH_HOME/bin/nutch updatedb $CRAWL_DB $s2
echo ""
$NUTCH_HOME/bin/nutch updatedb $CRAWL_DB $s2
echo $NUTCH_HOME/bin/nutch invertlinks $LINK_DB -dir $SEGMENTS_DIR
echo ""
$NUTCH_HOME/bin/nutch invertlinks $LINK_DB -dir $SEGMENTS_DIR
echo $NUTCH_HOME/bin/nutch solrindex http://localhost:8080/nutchsolr/
$CRAWLDB $LINK_DB -dir $SEGMENTS_DIR/*
echo ""
$NUTCH_HOME/bin/nutch solrindex http://localhost:8080/nutchsolr/
$CRAWLDB $LINK_DB -dir $SEGMENTS_DIR/*
echo "cleaning $SEGMENTS_DIR"
#rm -rf $SEGMENTS_DIR/*
echo "crawl.sh: FINISHED: Crawl completed!"
echo ""
-- Chris
Re: problem with tutorial
Posted by Christopher Gross <co...@gmail.com>.
./nutch org.apache.nutch.net.URLFilterChecker -allCombined < in.txt
Checking combination of all URLFilters available
+http://urlAlpha.docx
So it looks like it is a valid one, right? Any other testing tools to try?
-- Chris
On Mon, Dec 19, 2011 at 2:52 PM, Markus Jelsma
<ma...@openindex.io> wrote:
> You must feed URL's from stdin.
>
>> Does it normally take a long time to run? It's been going about 5
>> minutes...
>>
>> -- Chris
>>
>>
>>
>> On Mon, Dec 19, 2011 at 2:43 PM, Markus Jelsma
>>
>> <ma...@openindex.io> wrote:
>> >> bin/nutch org.apache.nutch.net.URLFilterChecker -allCombined
Re: problem with tutorial
Posted by Markus Jelsma <ma...@openindex.io>.
You must feed URL's from stdin.
> Does it normally take a long time to run? It's been going about 5
> minutes...
>
> -- Chris
>
>
>
> On Mon, Dec 19, 2011 at 2:43 PM, Markus Jelsma
>
> <ma...@openindex.io> wrote:
> >> bin/nutch org.apache.nutch.net.URLFilterChecker -allCombined
Re: problem with tutorial
Posted by Christopher Gross <co...@gmail.com>.
Does it normally take a long time to run? It's been going about 5 minutes...
-- Chris
On Mon, Dec 19, 2011 at 2:43 PM, Markus Jelsma
<ma...@openindex.io> wrote:
>
>> bin/nutch org.apache.nutch.net.URLFilterChecker -allCombined
>
Re: problem with tutorial
Posted by Markus Jelsma <ma...@openindex.io>.
> bin/nutch org.apache.nutch.net.URLFilterChecker -allCombined
Re: problem with tutorial
Posted by Christopher Gross <co...@gmail.com>.
The URLFilterChecker tool doesn't have a page yet...what is the syntax
& parameters for it?
-- Chris
On Mon, Dec 19, 2011 at 2:33 PM, Markus Jelsma
<ma...@openindex.io> wrote:
>
>> On Mon, Dec 19, 2011 at 2:17 PM, Markus Jelsma
>>
>> <ma...@openindex.io> wrote:
>> > What do you mean by skipping over? You don't want ppt pptx and things? In
>> > all cases you need to set up URL filters specific for your scenario and
>> > whishes.
>>
>> I want to index all the office type documents, they're getting skipped
>> over and I don't know why.
>>
>> I have altered the regex-urlfilter.xml to NOT remove those, but
>> they're still not getting crawled.
>
> You need to check all filters that are enabled through your plugin.includes.
> There's a org.apache.nutch.net.URLFilterChecker tool. It works a bit strange
> with with -allCombined switch you can make sure it passes your URL's or not.
>
>>
>> Thanks!
>>
>> -- Chris
Re: problem with tutorial
Posted by Markus Jelsma <ma...@openindex.io>.
> On Mon, Dec 19, 2011 at 2:17 PM, Markus Jelsma
>
> <ma...@openindex.io> wrote:
> > What do you mean by skipping over? You don't want ppt pptx and things? In
> > all cases you need to set up URL filters specific for your scenario and
> > whishes.
>
> I want to index all the office type documents, they're getting skipped
> over and I don't know why.
>
> I have altered the regex-urlfilter.xml to NOT remove those, but
> they're still not getting crawled.
You need to check all filters that are enabled through your plugin.includes.
There's a org.apache.nutch.net.URLFilterChecker tool. It works a bit strange
with with -allCombined switch you can make sure it passes your URL's or not.
>
> Thanks!
>
> -- Chris
Re: problem with tutorial
Posted by Christopher Gross <co...@gmail.com>.
On Mon, Dec 19, 2011 at 2:17 PM, Markus Jelsma
<ma...@openindex.io> wrote:
>
>
> What do you mean by skipping over? You don't want ppt pptx and things? In all
> cases you need to set up URL filters specific for your scenario and whishes.
I want to index all the office type documents, they're getting skipped
over and I don't know why.
I have altered the regex-urlfilter.xml to NOT remove those, but
they're still not getting crawled.
Thanks!
-- Chris
Re: problem with tutorial
Posted by Markus Jelsma <ma...@openindex.io>.
> I figured that it would be better to do it that way -- but I'm having
> issues getting them to work, which may be caused by my configurations,
> but I'm not able to find good help trying to get them figured out
> either. I've been trying to crawl a SharePoint site (anonymous access
> allowed, so not even bothering with authentication) for over a week
> and not really getting any farther along. It can find the directories
> and aspx pages and some of the pdfs, but then it skips over other
> pdfs, as well as office docs (doc, docx, ppt, pptx, etc.) I'm
> guessing that is Tika related, but I'm not seeing much on that either.
What do you mean by skipping over? You don't want ppt pptx and things? In all
cases you need to set up URL filters specific for your scenario and whishes.
>
> I don't get what the plugin.includes value even does, and I can't find
> an explanation of that on the wiki.
All configuration directives have a description, see nutch-default.xml. It's
wise to go through all settings. You won't understand them all at once but
later on you will remember them for a new problem you encouter.
> Is there a published book
> somewhere, or some more thorough documentation?
No. The documentation on the wiki and internet tutorials will not tell you how
crawling actually works and what's involved.
> I'm just getting
> frustrated and running in circles trying to make sense of all this.
>
> Can you give me a good starting point, or a good reference to use?
Well, yes, the nutch-default descriptions and unit tests and the rest of the
sources.
>
> -- Chris
>
>
>
> On Mon, Dec 19, 2011 at 1:59 PM, Markus Jelsma
>
> <ma...@openindex.io> wrote:
> > I prefer and encourage use of individual commands because: a) you learn
> > how the tools actually work and get a better understanding and b) the
> > crawl command does not expose all options of separate commands.
> >
> >> I created a script based on:
> >> http://wiki.apache.org/nutch/NutchTutorial#A3.2_Using_Individual_Command
> >> s_f or_Whole-Web_Crawling
> >>
> >> The script is similar to what happens in the old runbot.sh script, but
> >> it isn't working for me. The part that does the "s1" barely gets
> >> anything, but then the "s2" fails completely (it tries to make the
> >> same segment?).
> >>
> >> The plain "crawl" command seems to work, but how practical is it to
> >> use that instead of making this script? Is it better to have all the
> >> parts broken down like this, instead of passing more parts to the
> >> "crawl" command and letting that handle things?
> >>
> >> Any help would be greatly appreciated!
> >>
> >> Script:
> >>
> >> # runbot script to run the Nutch bot for crawling and re-crawling.
> >> # Usage: bin/runbot [safe]
> >> # If executed in 'safe' mode, it doesn't delete the temporary
> >> # directories generated during crawl. This might be helpful for
> >> # analysis and recovery in case a crawl fails.
> >> #
> >> # Author: Susam Pal
> >>
> >> #Set up some local vars -- like NUTCH_HOME & CATALINA_HOME
> >> . /common/setenv.sh
> >>
> >> CRAWL_DIR=$NUTCH_HOME/crawl
> >> SEGMENTS_DIR=$CRAWL_DIR/segments
> >> CRAWL_DB=$CRAWL_DIR/crawldb
> >> LINK_DB=$CRAWL_DIR/linkdb
> >>
> >> depth=5
> >> threads=5
> >> adddays=5
> >> topN=15 #Comment this statement if you don't want to set topN value
> >>
> >> # Arguments for rm and mv
> >> RMARGS="-rf"
> >> MVARGS="--verbose"
> >>
> >> # Parse arguments
> >> if [ "$1" == "safe" ]
> >> then
> >> safe=yes
> >> fi
> >>
> >> if [ -z "$NUTCH_HOME" ]
> >> then
> >> NUTCH_HOME=/cdda/nutch
> >> echo runbot: $0 could not find environment variable NUTCH_HOME
> >> echo runbot: NUTCH_HOME=$NUTCH_HOME has been set by the script
> >> else
> >> echo runbot: $0 found environment variable NUTCH_HOME=$NUTCH_HOME
> >> fi
> >>
> >> if [ -z "$CATALINA_HOME" ]
> >> then
> >> CATALINA_HOME=/cdda/apache-tomcat
> >> echo runbot: $0 could not find environment variable NUTCH_HOME
> >> echo runbot: CATALINA_HOME=$CATALINA_HOME has been set by the script
> >> else
> >> echo runbot: $0 found environment variable
> >> CATALINA_HOME=$CATALINA_HOME fi
> >>
> >> if [ -n "$topN" ]
> >> then
> >> topN="-topN $topN"
> >> else
> >> topN=""
> >> fi
> >>
> >> echo "crawl.sh: STARTING"
> >> echo $NUTCH_HOME/bin/nutch inject $CRAWL_DB $NUTCH_HOME/urls
> >> echo ""
> >> $NUTCH_HOME/bin/nutch inject $CRAWL_DB $NUTCH_HOME/urls
> >>
> >> echo $NUTCH_HOME/bin/nutch generate $NUTCH_HOME/crawl/crawldb
> >> $SEGMENTS_DIR -numFetchers 5
> >> echo ""
> >> $NUTCH_HOME/bin/nutch generate $CRAWL_DB $SEGMENTS_DIR -numFetchers 5
> >>
> >> s1=`ls -rd $SEGMENTS_DIR/* | tail -1`
> >> echo Segments: $s1
> >> echo ""
> >>
> >> echo $NUTCH_HOME/bin/nutch fetch $s1 -threads $threads
> >> echo ""
> >> $NUTCH_HOME/bin/nutch fetch $s1 -threads $threads
> >>
> >> echo $NUTCH_HOME/bin/nutch parse $s1
> >> echo ""
> >> $NUTCH_HOME/bin/nutch parse $s1
> >>
> >> echo $NUTCH_HOME/bin/nutch updatedb $CRAWL_DB $s1
> >> echo ""
> >> $NUTCH_HOME/bin/nutch updatedb $CRAWL_DB $s1
> >>
> >> echo $NUTCH_HOME/bin/nutch generate crawldb $SEGMENTS_DIR -numFetchers 5
> >> echo ""
> >> $NUTCH_HOME/bin/nutch generate crawldb $SEGMENTS_DIR -numFetchers 5
> >> s2=`ls -rd $SEGMENTS_DIR/* | tail -1`
> >> echo Segments: $s2
> >> echo ""
> >>
> >> echo $NUTCH_HOME/bin/nutch fetch $s2 -threads $threads
> >> echo ""
> >> $NUTCH_HOME/bin/nutch fetch $s2 -threads $threads
> >>
> >> echo $NUTCH_HOME/bin/nutch parse $s2
> >> echo ""
> >> $NUTCH_HOME/bin/nutch parse $s2
> >>
> >> echo $NUTCH_HOME/bin/nutch updatedb $CRAWL_DB $s2
> >> echo ""
> >> $NUTCH_HOME/bin/nutch updatedb $CRAWL_DB $s2
> >>
> >> echo $NUTCH_HOME/bin/nutch invertlinks $LINK_DB -dir $SEGMENTS_DIR
> >> echo ""
> >> $NUTCH_HOME/bin/nutch invertlinks $LINK_DB -dir $SEGMENTS_DIR
> >>
> >> echo $NUTCH_HOME/bin/nutch solrindex http://localhost:8080/nutchsolr/
> >> $CRAWLDB $LINK_DB -dir $SEGMENTS_DIR/*
> >> echo ""
> >> $NUTCH_HOME/bin/nutch solrindex http://localhost:8080/nutchsolr/
> >> $CRAWLDB $LINK_DB -dir $SEGMENTS_DIR/*
> >>
> >> echo "cleaning $SEGMENTS_DIR"
> >> #rm -rf $SEGMENTS_DIR/*
> >>
> >> echo "crawl.sh: FINISHED: Crawl completed!"
> >> echo ""
> >>
> >>
> >>
> >> -- Chris
Re: problem with tutorial
Posted by Christopher Gross <co...@gmail.com>.
I figured that it would be better to do it that way -- but I'm having
issues getting them to work, which may be caused by my configurations,
but I'm not able to find good help trying to get them figured out
either. I've been trying to crawl a SharePoint site (anonymous access
allowed, so not even bothering with authentication) for over a week
and not really getting any farther along. It can find the directories
and aspx pages and some of the pdfs, but then it skips over other
pdfs, as well as office docs (doc, docx, ppt, pptx, etc.) I'm
guessing that is Tika related, but I'm not seeing much on that either.
I don't get what the plugin.includes value even does, and I can't find
an explanation of that on the wiki. Is there a published book
somewhere, or some more thorough documentation? I'm just getting
frustrated and running in circles trying to make sense of all this.
Can you give me a good starting point, or a good reference to use?
-- Chris
On Mon, Dec 19, 2011 at 1:59 PM, Markus Jelsma
<ma...@openindex.io> wrote:
> I prefer and encourage use of individual commands because: a) you learn how
> the tools actually work and get a better understanding and b) the crawl
> command does not expose all options of separate commands.
>
>> I created a script based on:
>> http://wiki.apache.org/nutch/NutchTutorial#A3.2_Using_Individual_Commands_f
>> or_Whole-Web_Crawling
>>
>> The script is similar to what happens in the old runbot.sh script, but
>> it isn't working for me. The part that does the "s1" barely gets
>> anything, but then the "s2" fails completely (it tries to make the
>> same segment?).
>>
>> The plain "crawl" command seems to work, but how practical is it to
>> use that instead of making this script? Is it better to have all the
>> parts broken down like this, instead of passing more parts to the
>> "crawl" command and letting that handle things?
>>
>> Any help would be greatly appreciated!
>>
>> Script:
>>
>> # runbot script to run the Nutch bot for crawling and re-crawling.
>> # Usage: bin/runbot [safe]
>> # If executed in 'safe' mode, it doesn't delete the temporary
>> # directories generated during crawl. This might be helpful for
>> # analysis and recovery in case a crawl fails.
>> #
>> # Author: Susam Pal
>>
>> #Set up some local vars -- like NUTCH_HOME & CATALINA_HOME
>> . /common/setenv.sh
>>
>> CRAWL_DIR=$NUTCH_HOME/crawl
>> SEGMENTS_DIR=$CRAWL_DIR/segments
>> CRAWL_DB=$CRAWL_DIR/crawldb
>> LINK_DB=$CRAWL_DIR/linkdb
>>
>> depth=5
>> threads=5
>> adddays=5
>> topN=15 #Comment this statement if you don't want to set topN value
>>
>> # Arguments for rm and mv
>> RMARGS="-rf"
>> MVARGS="--verbose"
>>
>> # Parse arguments
>> if [ "$1" == "safe" ]
>> then
>> safe=yes
>> fi
>>
>> if [ -z "$NUTCH_HOME" ]
>> then
>> NUTCH_HOME=/cdda/nutch
>> echo runbot: $0 could not find environment variable NUTCH_HOME
>> echo runbot: NUTCH_HOME=$NUTCH_HOME has been set by the script
>> else
>> echo runbot: $0 found environment variable NUTCH_HOME=$NUTCH_HOME
>> fi
>>
>> if [ -z "$CATALINA_HOME" ]
>> then
>> CATALINA_HOME=/cdda/apache-tomcat
>> echo runbot: $0 could not find environment variable NUTCH_HOME
>> echo runbot: CATALINA_HOME=$CATALINA_HOME has been set by the script
>> else
>> echo runbot: $0 found environment variable CATALINA_HOME=$CATALINA_HOME
>> fi
>>
>> if [ -n "$topN" ]
>> then
>> topN="-topN $topN"
>> else
>> topN=""
>> fi
>>
>> echo "crawl.sh: STARTING"
>> echo $NUTCH_HOME/bin/nutch inject $CRAWL_DB $NUTCH_HOME/urls
>> echo ""
>> $NUTCH_HOME/bin/nutch inject $CRAWL_DB $NUTCH_HOME/urls
>>
>> echo $NUTCH_HOME/bin/nutch generate $NUTCH_HOME/crawl/crawldb
>> $SEGMENTS_DIR -numFetchers 5
>> echo ""
>> $NUTCH_HOME/bin/nutch generate $CRAWL_DB $SEGMENTS_DIR -numFetchers 5
>>
>> s1=`ls -rd $SEGMENTS_DIR/* | tail -1`
>> echo Segments: $s1
>> echo ""
>>
>> echo $NUTCH_HOME/bin/nutch fetch $s1 -threads $threads
>> echo ""
>> $NUTCH_HOME/bin/nutch fetch $s1 -threads $threads
>>
>> echo $NUTCH_HOME/bin/nutch parse $s1
>> echo ""
>> $NUTCH_HOME/bin/nutch parse $s1
>>
>> echo $NUTCH_HOME/bin/nutch updatedb $CRAWL_DB $s1
>> echo ""
>> $NUTCH_HOME/bin/nutch updatedb $CRAWL_DB $s1
>>
>> echo $NUTCH_HOME/bin/nutch generate crawldb $SEGMENTS_DIR -numFetchers 5
>> echo ""
>> $NUTCH_HOME/bin/nutch generate crawldb $SEGMENTS_DIR -numFetchers 5
>> s2=`ls -rd $SEGMENTS_DIR/* | tail -1`
>> echo Segments: $s2
>> echo ""
>>
>> echo $NUTCH_HOME/bin/nutch fetch $s2 -threads $threads
>> echo ""
>> $NUTCH_HOME/bin/nutch fetch $s2 -threads $threads
>>
>> echo $NUTCH_HOME/bin/nutch parse $s2
>> echo ""
>> $NUTCH_HOME/bin/nutch parse $s2
>>
>> echo $NUTCH_HOME/bin/nutch updatedb $CRAWL_DB $s2
>> echo ""
>> $NUTCH_HOME/bin/nutch updatedb $CRAWL_DB $s2
>>
>> echo $NUTCH_HOME/bin/nutch invertlinks $LINK_DB -dir $SEGMENTS_DIR
>> echo ""
>> $NUTCH_HOME/bin/nutch invertlinks $LINK_DB -dir $SEGMENTS_DIR
>>
>> echo $NUTCH_HOME/bin/nutch solrindex http://localhost:8080/nutchsolr/
>> $CRAWLDB $LINK_DB -dir $SEGMENTS_DIR/*
>> echo ""
>> $NUTCH_HOME/bin/nutch solrindex http://localhost:8080/nutchsolr/
>> $CRAWLDB $LINK_DB -dir $SEGMENTS_DIR/*
>>
>> echo "cleaning $SEGMENTS_DIR"
>> #rm -rf $SEGMENTS_DIR/*
>>
>> echo "crawl.sh: FINISHED: Crawl completed!"
>> echo ""
>>
>>
>>
>> -- Chris
Re: problem with tutorial
Posted by Markus Jelsma <ma...@openindex.io>.
I prefer and encourage use of individual commands because: a) you learn how
the tools actually work and get a better understanding and b) the crawl
command does not expose all options of separate commands.
> I created a script based on:
> http://wiki.apache.org/nutch/NutchTutorial#A3.2_Using_Individual_Commands_f
> or_Whole-Web_Crawling
>
> The script is similar to what happens in the old runbot.sh script, but
> it isn't working for me. The part that does the "s1" barely gets
> anything, but then the "s2" fails completely (it tries to make the
> same segment?).
>
> The plain "crawl" command seems to work, but how practical is it to
> use that instead of making this script? Is it better to have all the
> parts broken down like this, instead of passing more parts to the
> "crawl" command and letting that handle things?
>
> Any help would be greatly appreciated!
>
> Script:
>
> # runbot script to run the Nutch bot for crawling and re-crawling.
> # Usage: bin/runbot [safe]
> # If executed in 'safe' mode, it doesn't delete the temporary
> # directories generated during crawl. This might be helpful for
> # analysis and recovery in case a crawl fails.
> #
> # Author: Susam Pal
>
> #Set up some local vars -- like NUTCH_HOME & CATALINA_HOME
> . /common/setenv.sh
>
> CRAWL_DIR=$NUTCH_HOME/crawl
> SEGMENTS_DIR=$CRAWL_DIR/segments
> CRAWL_DB=$CRAWL_DIR/crawldb
> LINK_DB=$CRAWL_DIR/linkdb
>
> depth=5
> threads=5
> adddays=5
> topN=15 #Comment this statement if you don't want to set topN value
>
> # Arguments for rm and mv
> RMARGS="-rf"
> MVARGS="--verbose"
>
> # Parse arguments
> if [ "$1" == "safe" ]
> then
> safe=yes
> fi
>
> if [ -z "$NUTCH_HOME" ]
> then
> NUTCH_HOME=/cdda/nutch
> echo runbot: $0 could not find environment variable NUTCH_HOME
> echo runbot: NUTCH_HOME=$NUTCH_HOME has been set by the script
> else
> echo runbot: $0 found environment variable NUTCH_HOME=$NUTCH_HOME
> fi
>
> if [ -z "$CATALINA_HOME" ]
> then
> CATALINA_HOME=/cdda/apache-tomcat
> echo runbot: $0 could not find environment variable NUTCH_HOME
> echo runbot: CATALINA_HOME=$CATALINA_HOME has been set by the script
> else
> echo runbot: $0 found environment variable CATALINA_HOME=$CATALINA_HOME
> fi
>
> if [ -n "$topN" ]
> then
> topN="-topN $topN"
> else
> topN=""
> fi
>
> echo "crawl.sh: STARTING"
> echo $NUTCH_HOME/bin/nutch inject $CRAWL_DB $NUTCH_HOME/urls
> echo ""
> $NUTCH_HOME/bin/nutch inject $CRAWL_DB $NUTCH_HOME/urls
>
> echo $NUTCH_HOME/bin/nutch generate $NUTCH_HOME/crawl/crawldb
> $SEGMENTS_DIR -numFetchers 5
> echo ""
> $NUTCH_HOME/bin/nutch generate $CRAWL_DB $SEGMENTS_DIR -numFetchers 5
>
> s1=`ls -rd $SEGMENTS_DIR/* | tail -1`
> echo Segments: $s1
> echo ""
>
> echo $NUTCH_HOME/bin/nutch fetch $s1 -threads $threads
> echo ""
> $NUTCH_HOME/bin/nutch fetch $s1 -threads $threads
>
> echo $NUTCH_HOME/bin/nutch parse $s1
> echo ""
> $NUTCH_HOME/bin/nutch parse $s1
>
> echo $NUTCH_HOME/bin/nutch updatedb $CRAWL_DB $s1
> echo ""
> $NUTCH_HOME/bin/nutch updatedb $CRAWL_DB $s1
>
> echo $NUTCH_HOME/bin/nutch generate crawldb $SEGMENTS_DIR -numFetchers 5
> echo ""
> $NUTCH_HOME/bin/nutch generate crawldb $SEGMENTS_DIR -numFetchers 5
> s2=`ls -rd $SEGMENTS_DIR/* | tail -1`
> echo Segments: $s2
> echo ""
>
> echo $NUTCH_HOME/bin/nutch fetch $s2 -threads $threads
> echo ""
> $NUTCH_HOME/bin/nutch fetch $s2 -threads $threads
>
> echo $NUTCH_HOME/bin/nutch parse $s2
> echo ""
> $NUTCH_HOME/bin/nutch parse $s2
>
> echo $NUTCH_HOME/bin/nutch updatedb $CRAWL_DB $s2
> echo ""
> $NUTCH_HOME/bin/nutch updatedb $CRAWL_DB $s2
>
> echo $NUTCH_HOME/bin/nutch invertlinks $LINK_DB -dir $SEGMENTS_DIR
> echo ""
> $NUTCH_HOME/bin/nutch invertlinks $LINK_DB -dir $SEGMENTS_DIR
>
> echo $NUTCH_HOME/bin/nutch solrindex http://localhost:8080/nutchsolr/
> $CRAWLDB $LINK_DB -dir $SEGMENTS_DIR/*
> echo ""
> $NUTCH_HOME/bin/nutch solrindex http://localhost:8080/nutchsolr/
> $CRAWLDB $LINK_DB -dir $SEGMENTS_DIR/*
>
> echo "cleaning $SEGMENTS_DIR"
> #rm -rf $SEGMENTS_DIR/*
>
> echo "crawl.sh: FINISHED: Crawl completed!"
> echo ""
>
>
>
> -- Chris