You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by Erik Hatcher <er...@ehatchersolutions.com> on 2005/08/18 20:44:50 UTC

Outlink metadata?

First a question about the current behavior... does Nutch adhere to  
the <a rel="nofollow"...> conventions?  If so, where is that coded?

On a related note, it seems carrying metadata around on Outlink would  
be beneficial, not just anchor text and URL.  For example, my  
application will crawl HTML sites with a HEAD <link> to RDF data.   
I'd like to, in an HtmlParseFilter, add ParseData metadata so that an  
indexer (a custom one currently, not the Nutch one) can get at the  
RDF data that has been fetched by the URL stored in the metadata.   
Make sense?

Would my use indicate that Outlink should carry along metadata or is  
there another way to achieve this (besides writing a custom HTML  
parser)?

Thanks,
     Erik