Need pointers regarding accessing crawled data/customizing policy for crawl.

[Manoj Bist <ma...@gmail.com>]

Hi,

I posted this on nutch-user earlier but it did not elicit any response. I
would really appreciate any pointers regarding this.

1.)  Is it possible to have a control on the 'policy'  to decide how soon a
url is fetched. For e.g. if a document does not change frequently, I would
like to fetch it less frequently etc.

2.) Is there an iterator for all the fetched urls?  I want to do operations
like the following:

   2.1     for each url in crawled urls {
              get content of the url and process it.
        }

   2.2. Access the content of the url directly i.e. given url 'u', access
the content of 'u'.


Thanks,

Manoj.
-- 
Tired of reading blogs? Listen to  your favorite blogs at
http://www.blogbard.com   !!!!

Re: Need pointers regarding accessing crawled data/customizing policy for crawl.

[Andrzej Bialecki <ab...@getopt.org>]

Manoj Bist wrote:
> Hi,
> 
> I posted this on nutch-user earlier but it did not elicit any response. I
> would really appreciate any pointers regarding this.
> 
> 1.)  Is it possible to have a control on the 'policy'  to decide how soon a
> url is fetched. For e.g. if a document does not change frequently, I would
> like to fetch it less frequently etc.

This is implemented in AdaptiveFetchSchedule - please see the javadoc 
for details.

> 
> 2.) Is there an iterator for all the fetched urls?  I want to do operations
> like the following:
> 
>    2.1     for each url in crawled urls {
>               get content of the url and process it.
>         }

Content is not stored in a single place, rather it's kept in a series of 
segments (shards), with possible duplicates. When you build an index out 
of a set of segments, and then de-duplicate this index, you can iterate 
over documents in this index - each document will contain unique 
pointers to <docNum, segmentName>, which you can then retrieve using 
MapFileOutputFormat.getReaders / getEntry (or use the SegmentReader API).


> 
>    2.2. Access the content of the url directly i.e. given url 'u', access
> the content of 'u'.

Please see the above, and especially the SegmentReader.get() method.


-- 
Best regards,
Andrzej Bialecki     <><
  ___. ___ ___ ___ _ _   __________________________________
[__ || __|__/|__||\/|  Information Retrieval, Semantic Web
___|||__||  \|  ||  |  Embedded Unix, System Integration
http://www.sigram.com  Contact: info at sigram dot com