Filtering webpages based on words / Fetch progress

[Mehdi Hemani <me...@gmail.com>]

Hi everybody,

This is my first e-mail at Nutch mailing list. I have two questions:

1. I want to filter out webpages based on a list of words. I have
tried filtering webpages based on url, but how to do it based on
words?

2. Is there a way to see the progress of Fetch process, like time
elapsed, time required to finish the process, what pages are being
currenlty fetched etc.?

Any help will be greatly appreciated.

Regards,
Mehdi

Re: Filtering webpages based on words / Fetch progress

[Lukas Vlcek <lu...@gmail.com>]

Hi,

On 6/8/06, Mehdi Hemani <me...@gmail.com> wrote:
> 2. Is there a way to see the progress of Fetch process, like time
> elapsed, time required to finish the process, what pages are being
> currenlty fetched etc.?

Try set the following in nutch-site.xml file:

<property>
  <name>fetcher.verbose</name>
  <value>true</value>
  <description>If true, fetcher will log more verbosely.</description>
</property>

Regards,
Lukas

Re: Filtering webpages based on words / Fetch progress

[Stefan Neufeind <ap...@stefan-neufeind.de>]

Lukas Vlcek wrote:
> Hi again,
> 
> On 6/8/06, Mehdi Hemani <me...@gmail.com> wrote:
>> 1. I want to filter out webpages based on a list of words. I have
>> tried filtering webpages based on url, but how to do it based on
>> words?
> 
> As for this question check the following link:
> http://wiki.apache.org/nutch/CommandLineOptions
> 
> As far as I know this prune tool should be available for nutch 0.8 as
> well (at least I can see the class to be included in source code so
> you should be able to call it).

Pruning with 0.8-dev works fine here. You give it a file with your 
"queries" and all matching pages will be pruned from the index. There is 
also a dryrun-option available - use that when building your queries :-)

Note that documents are only pruned from the index, not from segments or 
the crawldb! So upon re-indexing or running another crawler-round be 
sure to apply pruning again.

   Stefan

Re: Filtering webpages based on words / Fetch progress

[Lukas Vlcek <lu...@gmail.com>]

Hi again,

On 6/8/06, Mehdi Hemani <me...@gmail.com> wrote:
> 1. I want to filter out webpages based on a list of words. I have
> tried filtering webpages based on url, but how to do it based on
> words?

As for this question check the following link:
http://wiki.apache.org/nutch/CommandLineOptions

As far as I know this prune tool should be available for nutch 0.8 as
well (at least I can see the class to be included in source code so
you should be able to call it).

Regards,
Lukas