You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by "hugo.ma" <hu...@gmail.com> on 2012/08/22 16:19:10 UTC
Question about recrawl
Hi.
This is a conceptual question, i haven't tried yet.
The nutch version i am using is 2.0.
Suppose that a full crawl has been made and the nutch hsql database is
filled with all the data.
In my perception if i run nutch again, it should refetch all urls according
to the fetchSchedule rules.
The process responsible for marking the url's for fetching is the Generator
mapper but looking to the code i see
this:(org.apache.nutch.crawl.GeneratorMapper)
if (Mark.GENERATE_MARK.checkMark(page) != null) {
if (GeneratorJob.LOG.isDebugEnabled()) {
GeneratorJob.LOG.debug("Skipping " + url + "; already generated");
}
return;
}
The Generate_MARK is allways != null because after the first crawl the field
'MARKERS' of the database has:
__prsmrk__*1345638110-1053938230_gnmrk_*1345638110-1053938230_ftcmrk_*1345638110-1053938230.
The GenerateMark is allways present.
So, is my assumption correct or am i missing something?
--
View this message in context: http://lucene.472066.n3.nabble.com/Question-about-recrawl-tp4002651.html
Sent from the Nutch - User mailing list archive at Nabble.com.
Re: Question about recrawl
Posted by Ferdy Galema <fe...@kalooga.com>.
I do not have much experience with refreshing, but it seems this bug is
related to a markers bug in the DbUpdaterReducer. If you look in that class
(in HEAD) there is a line at 183 that should remove the generator mark..
On Wed, Aug 22, 2012 at 4:19 PM, hugo.ma <hu...@gmail.com> wrote:
> Hi.
>
> This is a conceptual question, i haven't tried yet.
> The nutch version i am using is 2.0.
>
> Suppose that a full crawl has been made and the nutch hsql database is
> filled with all the data.
> In my perception if i run nutch again, it should refetch all urls
> according
> to the fetchSchedule rules.
> The process responsible for marking the url's for fetching is the Generator
> mapper but looking to the code i see
> this:(org.apache.nutch.crawl.GeneratorMapper)
> if (Mark.GENERATE_MARK.checkMark(page) != null) {
> if (GeneratorJob.LOG.isDebugEnabled()) {
> GeneratorJob.LOG.debug("Skipping " + url + "; already generated");
> }
> return;
> }
>
> The Generate_MARK is allways != null because after the first crawl the
> field
> 'MARKERS' of the database has:
> __prsmrk__*1345638110-1053938230 _gnmrk_*1345638110-1053938230
> _ftcmrk_*1345638110-1053938230.
> The GenerateMark is allways present.
>
> So, is my assumption correct or am i missing something?
>
>
>
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Question-about-recrawl-tp4002651.html
> Sent from the Nutch - User mailing list archive at Nabble.com.
>