Additional URL Content

[John Martyniak <jo...@beforedawn.com>]

Hello everyone,

Part of the requirements for a site that I am working on is that I  
have some information in a DB and some in a nutch index.

The nutch index obviously contains the indexed URLs, etc.  However I  
also have a DB that contains the URLs and a bunch of other information  
about the URL, for example comments, ranking, etc.

What is the best way to update the DB based on what the spider finds.   
One approach I was thinking of was to not update the DB until somebody  
actually requests the data, and then add the URLs to the DB at that  
time.  It is kind of backwards especially if there is other data that  
needs to be collected from external sites, etc.

The second question is I have I would like to tag the URL that was  
fetched indexed, as a particular type, for example RSS feeds would get  
tagged one way, HTML another way.  So that when I perform a query  
against the Nutch index I can return only RSS, or return only Web.  Is  
the best way to do this some sort of tagging with a plugin?  Or can  
nutch perform that "out of the box"?

Thanks in advance for all of the advice.

-John