custom generator

[nishant rathore <ni...@gmail.com>]

Hi,

I am new to nutch and building an application that uses nutch to crawl
certain folders.
How can I make changes to generator configuration so that it picks the sub
directory for crawl only when the modified date of the directory is after
certain date.
Any pointers will be very helpful.


try {                         ToolRunner.run(NutchConfiguration.create(),
new Crawl(),                                         tokenize(crawlArg));
              } catch
(Exception<http://www.google.com/search?hl=en&q=allinurl%3Aexception+java.sun.com&btnI=I%27m%20Feeling%20Lucky>e
) {                         e.printStackTrace();
return;                 }


Thanks,
Nishant

Re: custom generator

[Lewis John Mcgibbney <le...@gmail.com>]

Hi Nishant,

On Sat, Jul 28, 2012 at 11:33 AM, nishant rathore
<ni...@gmail.com> wrote:

> I am new to nutch and building an application that uses nutch to crawl
> certain folders.

Do you plan to crawl a local file system or the web graph? In the
former AFAIK there is no way to tell whether one or more of the
contents within a folder (directory or subdirectory) have been
modified as there is no mechanism for tracking changes to directory
contents by the directory itself.

> How can I make changes to generator configuration so that it picks the sub
> directory for crawl only when the modified date of the directory is after
> certain date.

Lewis