You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Raghavendra Prabhu <rr...@gmail.com> on 2006/02/14 15:00:07 UTC
writing modified date in crawl datum
Hi
Currently the trunk has provision to setModified time in the crawldatum
This is not currently used . Will this be used in the future ?
Rgds
Prabhu
Re: writing modified date in crawl datum
Posted by Andrzej Bialecki <ab...@getopt.org>.
Raghavendra Prabhu wrote:
> Thanks Andrez
>
> One more thing which i wanted to know was fetching the last modified date is
> done as a part of content parsing function.
>
> Cant we split the whole thing into two funtions
> 1)The first function will getmodified date
> 2) The second function will get content
>
> Initiall the first function will be called to get the modified date and
> compared against crawldb modified date and only if it is different will the
> second function (which actually fetches the content) will be called
>
> Will this be the motive of adpative interval patch?
>
Yes, this is included in that patch.
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
Re: writing modified date in crawl datum
Posted by Raghavendra Prabhu <rr...@gmail.com>.
Thanks Andrez
One more thing which i wanted to know was fetching the last modified date is
done as a part of content parsing function.
Cant we split the whole thing into two funtions
1)The first function will getmodified date
2) The second function will get content
Initiall the first function will be called to get the modified date and
compared against crawldb modified date and only if it is different will the
second function (which actually fetches the content) will be called
Will this be the motive of adpative interval patch?
Rgds
Prabhu
On 2/14/06, Andrzej Bialecki <ab...@getopt.org> wrote:
>
> Raghavendra Prabhu wrote:
> > Hi
> >
> > Currently the trunk has provision to setModified time in the crawldatum
> >
> > This is not currently used . Will this be used in the future ?
> >
> >
>
> Yes, it was added in anticipation of the adaptive fetch interval
> patches. However, now I think it should probably be stored in the
> CrawlDatum metadata. I'm working on this patch, I'll update it soon on
> JIRA.
>
> --
> Best regards,
> Andrzej Bialecki <><
> ___. ___ ___ ___ _ _ __________________________________
> [__ || __|__/|__||\/| Information Retrieval, Semantic Web
> ___|||__|| \| || | Embedded Unix, System Integration
> http://www.sigram.com Contact: info at sigram dot com
>
>
>
Re: writing modified date in crawl datum
Posted by Andrzej Bialecki <ab...@getopt.org>.
Raghavendra Prabhu wrote:
> Hi
>
> Currently the trunk has provision to setModified time in the crawldatum
>
> This is not currently used . Will this be used in the future ?
>
>
Yes, it was added in anticipation of the adaptive fetch interval
patches. However, now I think it should probably be stored in the
CrawlDatum metadata. I'm working on this patch, I'll update it soon on JIRA.
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com