generating and updating segments

[Michael Coffey <mc...@yahoo.com.INVALID>]

In search of more effective parallelism, I have been experimenting with different schemes for organizing the nutch jobs. I would like to know if the Generator can work in a way that supports what I'm trying to do.
Here is a pseudocode description of one approach. I use variables named curSegs and prevSegs to refer to lists of segments. SegsPerWave is typically 4 or more.

prevSegs = generate( segsPerWave ) 
in a "background" process (on other machines):    fetch and parse prevSegs
repeat indefinitely    curSegs = generate( segsPerWave ) 
    in a "background" process (on other machines):        fetch and parse curSegs    wait for prevSegs to be fetched and parsed
    update, linkdb, and merge prevSegs    prevSegs = curSegs
As I understand it, this will not work right if I do not set generate.update.crawldb = true. In my subsequent calls to generate, it would generate duplicated (or partially duplicated) segments.
If I do set generate.update.crawldb = true, should it work right?  What, exactly, does generate.update.crawldb = true do? I assume it changes something in the crawldb, but I don't know what.