You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Sunnyvale Fl <su...@gmail.com> on 2010/02/02 23:30:06 UTC
Re: repeat fetch of same page without error
Looks like the right patch for my problem; unfortunately we are still on
Nutch 0.9. The patches are for FetchSchedule which doesn't exist in 0.9...
Any idea? Is there an older patch?
Thanks!
On Thu, Jan 21, 2010 at 6:35 PM, reinhard schwab <re...@aon.at>wrote:
> some time ago i have had the same with nutch 1.0 and i have discovered
> one bug.
>
> https://issues.apache.org/jira/browse/NUTCH-774
> https://issues.apache.org/jira/browse/NUTCH-773
>
> you will find patches there.
>
> Sunnyvale Fl schrieb:
> > You know you are right. I dump db for another url and the retry interval
> is
> > 0.0. For the same crawl, some url's retry interval is 7.0. Why is that?
> I
> > have db.default.fetch.interval set to 7.0 in nutch-site.xml. Thanks!
> >
> > Version: 5
> > Status: 2 (db_fetched)
> > Fetch time: Thu Jan 21 08:55:24 PST 2010
> > Modified time: Wed Dec 31 16:00:00 PST 1969
> > Retries since fetch: 0
> > Retry interval: 0.0 days
> > Score: 0.0
> > Signature: 09854146546e5e7fe5def1e1add23037
> > Metadata: _pst_:success(1), lastModified=0
> >
> >
> > On Thu, Jan 21, 2010 at 5:50 PM, reinhard schwab <reinhard.schwab@aon.at
> >wrote:
> >
> >
> >> yes, i mean that.
> >> in the java classes, it is called fetch interval, see CrawlDatum class.
> >> do you use the adddays option when generating the segment?
> >> if the value is higher than the fetch interval, then it can also happen
> >> that you
> >> crawl again and again a page.
> >>
> >> the fetch time in your entry is Nov 06 2009.
> >> the last time it has been fetched is before this date.
> >> it has not been refetched since that time.
> >>
> >>
> >> Sunnyvale Fl schrieb:
> >>
> >>> You mean the retry interval? It is 7 days from readdb -
> >>>
> >>> Version: 5
> >>> Status: 2 (db_fetched)
> >>> Fetch time: Fri Nov 06 07:48:54 PST 2009
> >>> Modified time: Wed Dec 31 16:00:00 PST 1969
> >>> Retries since fetch: 0
> >>> Retry interval: 7.0 days
> >>> Score: 0.0
> >>> Signature: 5ec8dc313a9ae4d61c6e8c9d9c18ea26
> >>> Metadata: _pst_:success(1), lastModified=0
> >>>
> >>>
> >>> On Thu, Jan 21, 2010 at 5:00 PM, reinhard schwab <
> reinhard.schwab@aon.at
> >>> wrote:
> >>>
> >>>
> >>>
> >>>> using "nutch readdb" you can dump the entry of the page.
> >>>> i believe that the fetch interval of this page is zero.
> >>>>
> >>>> Sunnyvale Fl schrieb:
> >>>>
> >>>>
> >>>>> Hi,
> >>>>> I am using Nutch 0.9.1 and I am having this weird problem - it will
> >>>>> repeatedly fetch the same page without error. So if I let it run to
> 10
> >>>>> levels deep, the same page will be fetched 10 times. What's wrong?
> >>>>>
> >>>>>
> >>>> Thanks!
> >>>>
> >>>>
> >>>
> >>
> >
> >
>
>
Re: repeat fetch of same page without error
Posted by Sunnyvale Fl <su...@gmail.com>.
seems to work!! thanks a lot for the help!!!
On Tue, Feb 2, 2010 at 5:07 PM, reinhard schwab <re...@aon.at>wrote:
> i have never used and tested 0.9.
> i have looked into the code, it is quite different to 1.0 in regard to
> CrawlDbReducer and scheduling.
> i propose to change the method
>
> public void setNextFetchTime() {
> fetchTime += (long)(MILLISECONDS_PER_DAY*fetchInterval);
> }
>
> in CrawlDatum.java
> check there for a fetchInterval with value 0.0. and set it to the
> default interval.
>
>
> Sunnyvale Fl schrieb:
> > Looks like the right patch for my problem; unfortunately we are still on
> > Nutch 0.9. The patches are for FetchSchedule which doesn't exist in
> 0.9...
> > Any idea? Is there an older patch?
> > Thanks!
> >
> > On Thu, Jan 21, 2010 at 6:35 PM, reinhard schwab <reinhard.schwab@aon.at
> >wrote:
> >
> >
> >> some time ago i have had the same with nutch 1.0 and i have discovered
> >> one bug.
> >>
> >> https://issues.apache.org/jira/browse/NUTCH-774
> >> https://issues.apache.org/jira/browse/NUTCH-773
> >>
> >> you will find patches there.
> >>
> >> Sunnyvale Fl schrieb:
> >>
> >>> You know you are right. I dump db for another url and the retry
> interval
> >>>
> >> is
> >>
> >>> 0.0. For the same crawl, some url's retry interval is 7.0. Why is
> that?
> >>>
> >> I
> >>
> >>> have db.default.fetch.interval set to 7.0 in nutch-site.xml. Thanks!
> >>>
> >>> Version: 5
> >>> Status: 2 (db_fetched)
> >>> Fetch time: Thu Jan 21 08:55:24 PST 2010
> >>> Modified time: Wed Dec 31 16:00:00 PST 1969
> >>> Retries since fetch: 0
> >>> Retry interval: 0.0 days
> >>> Score: 0.0
> >>> Signature: 09854146546e5e7fe5def1e1add23037
> >>> Metadata: _pst_:success(1), lastModified=0
> >>>
> >>>
> >>> On Thu, Jan 21, 2010 at 5:50 PM, reinhard schwab <
> reinhard.schwab@aon.at
> >>> wrote:
> >>>
> >>>
> >>>
> >>>> yes, i mean that.
> >>>> in the java classes, it is called fetch interval, see CrawlDatum
> class.
> >>>> do you use the adddays option when generating the segment?
> >>>> if the value is higher than the fetch interval, then it can also
> happen
> >>>> that you
> >>>> crawl again and again a page.
> >>>>
> >>>> the fetch time in your entry is Nov 06 2009.
> >>>> the last time it has been fetched is before this date.
> >>>> it has not been refetched since that time.
> >>>>
> >>>>
> >>>> Sunnyvale Fl schrieb:
> >>>>
> >>>>
> >>>>> You mean the retry interval? It is 7 days from readdb -
> >>>>>
> >>>>> Version: 5
> >>>>> Status: 2 (db_fetched)
> >>>>> Fetch time: Fri Nov 06 07:48:54 PST 2009
> >>>>> Modified time: Wed Dec 31 16:00:00 PST 1969
> >>>>> Retries since fetch: 0
> >>>>> Retry interval: 7.0 days
> >>>>> Score: 0.0
> >>>>> Signature: 5ec8dc313a9ae4d61c6e8c9d9c18ea26
> >>>>> Metadata: _pst_:success(1), lastModified=0
> >>>>>
> >>>>>
> >>>>> On Thu, Jan 21, 2010 at 5:00 PM, reinhard schwab <
> >>>>>
> >> reinhard.schwab@aon.at
> >>
> >>>>> wrote:
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>> using "nutch readdb" you can dump the entry of the page.
> >>>>>> i believe that the fetch interval of this page is zero.
> >>>>>>
> >>>>>> Sunnyvale Fl schrieb:
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>> Hi,
> >>>>>>> I am using Nutch 0.9.1 and I am having this weird problem - it will
> >>>>>>> repeatedly fetch the same page without error. So if I let it run
> to
> >>>>>>>
> >> 10
> >>
> >>>>>>> levels deep, the same page will be fetched 10 times. What's wrong?
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>> Thanks!
> >>>>>>
> >>>>>>
> >>>>>>
> >>>
> >>
> >
> >
>
>
Re: repeat fetch of same page without error
Posted by reinhard schwab <re...@aon.at>.
i have never used and tested 0.9.
i have looked into the code, it is quite different to 1.0 in regard to
CrawlDbReducer and scheduling.
i propose to change the method
public void setNextFetchTime() {
fetchTime += (long)(MILLISECONDS_PER_DAY*fetchInterval);
}
in CrawlDatum.java
check there for a fetchInterval with value 0.0. and set it to the
default interval.
Sunnyvale Fl schrieb:
> Looks like the right patch for my problem; unfortunately we are still on
> Nutch 0.9. The patches are for FetchSchedule which doesn't exist in 0.9...
> Any idea? Is there an older patch?
> Thanks!
>
> On Thu, Jan 21, 2010 at 6:35 PM, reinhard schwab <re...@aon.at>wrote:
>
>
>> some time ago i have had the same with nutch 1.0 and i have discovered
>> one bug.
>>
>> https://issues.apache.org/jira/browse/NUTCH-774
>> https://issues.apache.org/jira/browse/NUTCH-773
>>
>> you will find patches there.
>>
>> Sunnyvale Fl schrieb:
>>
>>> You know you are right. I dump db for another url and the retry interval
>>>
>> is
>>
>>> 0.0. For the same crawl, some url's retry interval is 7.0. Why is that?
>>>
>> I
>>
>>> have db.default.fetch.interval set to 7.0 in nutch-site.xml. Thanks!
>>>
>>> Version: 5
>>> Status: 2 (db_fetched)
>>> Fetch time: Thu Jan 21 08:55:24 PST 2010
>>> Modified time: Wed Dec 31 16:00:00 PST 1969
>>> Retries since fetch: 0
>>> Retry interval: 0.0 days
>>> Score: 0.0
>>> Signature: 09854146546e5e7fe5def1e1add23037
>>> Metadata: _pst_:success(1), lastModified=0
>>>
>>>
>>> On Thu, Jan 21, 2010 at 5:50 PM, reinhard schwab <reinhard.schwab@aon.at
>>> wrote:
>>>
>>>
>>>
>>>> yes, i mean that.
>>>> in the java classes, it is called fetch interval, see CrawlDatum class.
>>>> do you use the adddays option when generating the segment?
>>>> if the value is higher than the fetch interval, then it can also happen
>>>> that you
>>>> crawl again and again a page.
>>>>
>>>> the fetch time in your entry is Nov 06 2009.
>>>> the last time it has been fetched is before this date.
>>>> it has not been refetched since that time.
>>>>
>>>>
>>>> Sunnyvale Fl schrieb:
>>>>
>>>>
>>>>> You mean the retry interval? It is 7 days from readdb -
>>>>>
>>>>> Version: 5
>>>>> Status: 2 (db_fetched)
>>>>> Fetch time: Fri Nov 06 07:48:54 PST 2009
>>>>> Modified time: Wed Dec 31 16:00:00 PST 1969
>>>>> Retries since fetch: 0
>>>>> Retry interval: 7.0 days
>>>>> Score: 0.0
>>>>> Signature: 5ec8dc313a9ae4d61c6e8c9d9c18ea26
>>>>> Metadata: _pst_:success(1), lastModified=0
>>>>>
>>>>>
>>>>> On Thu, Jan 21, 2010 at 5:00 PM, reinhard schwab <
>>>>>
>> reinhard.schwab@aon.at
>>
>>>>> wrote:
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>> using "nutch readdb" you can dump the entry of the page.
>>>>>> i believe that the fetch interval of this page is zero.
>>>>>>
>>>>>> Sunnyvale Fl schrieb:
>>>>>>
>>>>>>
>>>>>>
>>>>>>> Hi,
>>>>>>> I am using Nutch 0.9.1 and I am having this weird problem - it will
>>>>>>> repeatedly fetch the same page without error. So if I let it run to
>>>>>>>
>> 10
>>
>>>>>>> levels deep, the same page will be fetched 10 times. What's wrong?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> Thanks!
>>>>>>
>>>>>>
>>>>>>
>>>
>>
>
>