Nutch Recrawl

["Nemani, Raj" <Ra...@turner.com>]

Hi all,

I am using the following script to do my re-crawl.  This is basically a
slightly modified version of the script that is found here.
http://wiki.apache.org/nutch/Crawl  

I have a small site that I would like to crawl using this script may be
3 times a day on a windows server by scheduling the script  using the
windows scheduled task feature.  The actual plan is to create a windows
batch file and call the CygWin bash tool and supply my script to it as
shown below

	C:\cygwin\bin\bash.exe -l nutchrecrawl

Where  " nutchrecrawl" is the file containing the script below.

Is the overall approach I am taking correct to achieve my objective?
How does Nutch determine that there are old documents that have been
updated and hence need to be crawled.  

I also read that " db.fetch.interval.default" property controls how and
when Nutch decides to re-crawl an existing document and the default is
30 days.  

Let us say I  change " db.fetch.interval.default" value from 30 days to
say 20 minutes (12000 seconds).
Then I run the script the first time and index the results into
Solr/Lucene index.
Then I go to my site and make a text change to an existing page (already
indexed during the first script run) site and immediately re-run the
script and index the results again to Solr/Lucene index.  Assuming that
these steps happened within 20 minutes then I should not see the change
I made to the page in the index yet.  If I run the script again the
third time after 20 minutes has passed then I should see my change in
the index.  Is my understanding correct?  

I also read about -adddays argument that could be added to the
'generate' step.  How does this option work?

Sorry for long email.  But I wanted to make sure I provide all the
information to make it easy to understand the issue.  I really
appreciate your help.

Thanks
Raj

**************************************************************
depth=2
threads=50
adddays=0
#topN=15 #Comment this statement if you don't want to set topN value

# Arguments for rm and mv
RMARGS="-rf"
MVARGS="--verbose"

# Parse arguments
if [ "$1" == "safe" ]
then
  safe=yes
fi

if [ -z "$NUTCH_HOME" ]
then
  NUTCH_HOME=/cygdrive/c/users/rnemani.turner/nutch
  cd /cygdrive/c/users/rnemani.turner/nutch
  echo runbot: $0 could not find environment variable NUTCH_HOME
  echo runbot: NUTCH_HOME=$NUTCH_HOME has been set by the script 
else
  echo runbot: $0 found environment variable NUTCH_HOME=$NUTCH_HOME 
fi

if [ -n "$topN" ]
then
  topN="-topN $topN"
else
  topN=""
fi

steps=7
echo "----- Inject (Step 1 of $steps) -----"
$NUTCH_HOME/bin/nutch inject crawl/crawldb urls

echo "----- Generate, Fetch, Parse, Update (Step 2 of $steps) -----"
for((i=0; i < $depth; i++))
do
  echo "--- Beginning crawl at depth `expr $i + 1` of $depth ---"
  $NUTCH_HOME/bin/nutch generate crawl/crawldb crawl/segments  
  if [ $? -ne 0 ]
  then
    echo "runbot: Stopping at depth $depth. No more URLs to fetch."
    break
  fi
  segment=`ls -d crawl/segments/* | tail -1`

  $NUTCH_HOME/bin/nutch fetch $segment -threads $threads
  if [ $? -ne 0 ]
  then
    echo "runbot: fetch $segment at depth `expr $i + 1` failed."
    echo "runbot: Deleting segment $segment."
    rm $RMARGS $segment
    continue
  fi

  $NUTCH_HOME/bin/nutch updatedb crawl/crawldb $segment
done

echo "----- Merge Segments (Step 3 of $steps) -----"
$NUTCH_HOME/bin/nutch mergesegs crawl/MERGEDsegments crawl/segments/*
if [ "$safe" != "yes" ]
then
  rm $RMARGS crawl/segments
else
  rm $RMARGS crawl/BACKUPsegments
  mv $MVARGS crawl/segments crawl/BACKUPsegments
fi

mv $MVARGS crawl/MERGEDsegments crawl/segments

echo "----- Invert Links (Step 4 of $steps) -----"
$NUTCH_HOME/bin/nutch invertlinks crawl/linkdb crawl/segments/*

echo "----- Index (Step 5 of $steps) -----"
$NUTCH_HOME/bin/nutch index crawl/NEWindexes crawl/crawldb crawl/linkdb
\
    crawl/segments/*

echo "----- Dedup (Step 6 of $steps) -----"
$NUTCH_HOME/bin/nutch dedup crawl/NEWindexes

echo "----- Merge Indexes (Step 7 of $steps) -----"
$NUTCH_HOME/bin/nutch merge crawl/NEWindex crawl/NEWindexes

#echo "----- Loading New Index (Step 8 of $steps) -----"

if [ "$safe" != "yes" ]
then
  rm $RMARGS crawl/NEWindexes
  rm $RMARGS crawl/index
else
  rm $RMARGS crawl/BACKUPindexes
  rm $RMARGS crawl/BACKUPindex
  mv $MVARGS crawl/NEWindexes crawl/BACKUPindexes
  mv $MVARGS crawl/index crawl/BACKUPindex
fi

mv $MVARGS crawl/NEWindex crawl/index

echo "runbot: FINISHED: Crawl completed!"
echo ""

******************************************************************