Incremental indexing

[Enzo Michelangeli <en...@gmail.com>]

As the size of my data keeps growing, and the indexing time grows even
faster, I'm trying to switch from a "reindex all at every crawl" model to an
incremental indexing one. I intend to keep the segments separate, but I
want to index only the segment fetched during the last cycle, and then merge
indexes and perhaps linkdb. I have a few questions:

1. In an incremental scenario, how do I remove from the indexes references
to segments that have expired??

2. Looking at http://wiki.apache.org/nutch/MergeCrawl , it would appear that
I can call "bin/nutch merge" with only two parameters: the original index
directory as destination, and the directory to be merged in the former:

  $nutch_dir/nutch merge $index_dir $new_indexes

But when I do that, the merged data are left in a subdirectory called 
$index_dir/merge_output . Shouldn't I instead create a new empty destination 
directory, do the merge, and then replace the original with the newly merged 
directory:

  merged_indexes=$crawl_dir/merged_indexes
  rm -rf $merged_indexes # just in case it's already there
  $nutch_dir/nutch merge $merged_indexes $index_dir $new_indexes
  rm -rf $index_dir.old # just in case it's already there
  mv $index_dir $index_dir.old
  mv $merged_indexes $index_dir
  rm -rf $index_dir.old

3. Regarding linkdb, does running "$nutch_dir/nutch invertlinks" on the 
latest segment only, and then merging the newly obtained linkdb with the 
current one with "$nutch_dir/nutch mergelinkdb", make sense rather than 
recreating linkdb afresh from the whole set of segments every time? In other 
words, can invertlinks work incrementally, or does it need to have a view of 
all segments in order to work correctly?

Thanks,

Enzo