metadata support in WebDB (Stefan's NUTCH-59 patch)

[og...@yahoo.com]

Hi Stefan,

http://issues.apache.org/jira/browse/NUTCH-59

This patch looks interesting for my Nutch needs, too.  ASF's SVN server
is down, and I don't have the Nutch repo on the computer I'm using, so
I can't look at the code, but looking at your diff, it looks like this
metadata would be stored somewhere inside Nutch's WebDB, and that one
would have to provide this metadata to Nutch during URL injection....
is this correct?

I currently have this little "wrapper method" around a few of Nutch's
tool classes (below).  I first generate a plain-text file with all URLs
I want to fetch, then I call the method below, and then I just call
Fetcher.main(...).  If I want to associate some metadata with each URL
to be fetched, where would I insert it into the system?  Would I need
my own injector class with my own addPage method that pulls metadata in
(from some external storage) for each URL it gets, and call
dbWriter.addPageIfNotPresent(page) like WebDBInjector does with DMOZ
data?

private void createFetchList(String urlFile) throws Exception {
  // create a Nutch WebDB
  String[] params;
  params = new String[] { _rootDir + "/db", "-create" };
  org.apache.nutch.tools.WebDBAdminTool.main(params);
 
  // inject URLs into the WebDB
  params = new String[] { _rootDir + "/db", "-urlfile", urlFile };
  org.apache.nutch.db.WebDBInjector.main(params);
 
  // create a fetch list for Nutch
  params = new String[] { _rootDir + "/db", _rootDir + "/segments" };
  org.apache.nutch.tools.FetchListTool.main(params);
}


Thanks,
Otis

Re: metadata support in WebDB (Stefan's NUTCH-59 patch)

[Stefan Groschupf <sg...@media-style.com>]

Hi Otis,

> http://issues.apache.org/jira/browse/NUTCH-59
>
> This patch looks interesting for my Nutch needs,
So please vote for the patch if you like it. :-)

> I can't look at the code, but looking at your diff, it looks like this
> metadata would be stored somewhere inside Nutch's WebDB, and that one
> would have to provide this metadata to Nutch during URL injection....
> is this correct?
Yes, meta data are part of the page object and stored in the webdb.
You can add  metadata in any situation you maintain this page object.
So you can have a custom injector as you describe to set meta data,  
but more interesting you can  set them until fetch time as well or in  
any situation you have access to the page object. (e.g. segment  
generation, dbupdated etc.)

>
> I currently have this little "wrapper method" around a few of Nutch's
> tool classes (below).  I first generate a plain-text file with all  
> URLs
> I want to fetch, then I call the method below, and then I just call
> Fetcher.main(...).  If I want to associate some metadata with each URL
> to be fetched, where would I insert it into the system?  Would I need
> my own injector class with my own addPage method that pulls  
> metadata in
> (from some external storage) for each URL it gets, and call
> dbWriter.addPageIfNotPresent(page) like WebDBInjector does with DMOZ
> data?

Yes.
I personal suggest create a extension point for the injector. May  
other people find that interesting as well and you can contribute  
this extension point.  :)
Write a small plugin that lookup the meta data you plan to add from a  
mysql-db or so and add them to the page object. That's it.
You can do very interesting things until the life-cycle of the page  
object.
For example generate metadata from html content, fetch time (more  
intelligent fetchnig), or hand over meta data from pages to links etc.

Keep in mind that you can have other storage types than the actually  
existing map for storing meta data. For example you can implement a  
StringArray or other datatypes.



In general I would love to see more interest in this patch and some  
votes since I think that such meta data can brings a lot of new  
possible features to nutch.
The very interesting part is that if you do not use meta data the web  
db is not blowed up and that this patch does not slow down the web db  
processing speed.

Greetings,
Stefan