Phasing out old segments

[ph...@comcast.net]

Hi All,
	I have been using nutch for a while now with a 300M+ urls  crawled in the 
last few months. So we have a lot of segments of mixed new data and recrawl 
data. While it is assumed safe to delete segments older than the refresh 
rate, it is not certain that all the urls in the old segments have been 
recrawled given the sheer number of urls that are in the database.
Some of the older segments also contain the top level homepages of many of the 
domains so I'd like to be sure that these are refreshed in another newer 
segment. 

Is anybody tackling this problem? 
If not, I have been thinking of building the following tool:

1) Read a segment or a collection of segments.
2) Compare each url to its entry in webdb:
	If the url was marked with -adddays from a previous fetchlist generation, 
ignore it.
	If the url was not accessible at last crawl, add it to list.
	If the url was last crawled with a 200 or 30? status longer than refresh rate 
days ago, add to list. (Not sure about 303 pages,  )	
3) sort the list and run it against the url filters.
4) generate a fetchlist with these urls. If there are no urls in this list, 
then this segment is ready for deletion. 
This part can be just a report on the state of the segments or can generate 
the segment without marking it in webdb.

I'd welcome any comment. 
If others find it useful, I will be happy to post it once it's done.

Phoebe.