You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by "Emmanuel Joke (JIRA)" <ji...@apache.org> on 2007/07/17 14:08:23 UTC
[jira] Created: (NUTCH-516) Next fetch time is not set when it is a
CrawlDatum.STATUS_FETCH_GONE
Next fetch time is not set when it is a CrawlDatum.STATUS_FETCH_GONE
--------------------------------------------------------------------
Key: NUTCH-516
URL: https://issues.apache.org/jira/browse/NUTCH-516
Project: Nutch
Issue Type: Bug
Components: fetcher
Environment: Java 1.6, Linux 2.6
Reporter: Emmanuel Joke
Fix For: 1.0.0
We can not crawl some page due to a robots restriction. In this case we update the db with the Metada: _pst_:robots_denied(18) , we add the status code 3 and we change the fecth interval to 67.5 days.
Unfortunetely the Fetch time is never change, so it keeps generating this page and fetching it every time.
We should update the schedule fetch in crawldb to reflect to the fetch interval.
We should add in crawldbreducer:
case CrawlDatum.STATUS_FETCH_GONE: // permanent failure
if (old != null)
result.setSignature(old.getSignature()); // use old signature
result.setStatus(CrawlDatum.STATUS_DB_GONE);
result = schedule.setPageGoneSchedule((Text)key, result, prevFetchTime,
prevModifiedTime, fetch.getFetchTime());
// set the schedule
result = schedule.setFetchSchedule((Text)key, result, prevFetchTime,
prevModifiedTime, fetch.getFetchTime(), fetch.getModifiedTime(), modified);
break;
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (NUTCH-516) Next fetch time is not set when it is
a CrawlDatum.STATUS_FETCH_GONE
Posted by "Doğacan Güney (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12513473 ]
Doğacan Güney commented on NUTCH-516:
-------------------------------------
Nice catch. I totally missed the int/long casting bug.
+1
> Next fetch time is not set when it is a CrawlDatum.STATUS_FETCH_GONE
> --------------------------------------------------------------------
>
> Key: NUTCH-516
> URL: https://issues.apache.org/jira/browse/NUTCH-516
> Project: Nutch
> Issue Type: Bug
> Components: fetcher
> Environment: Java 1.6, Linux 2.6
> Reporter: Emmanuel Joke
> Fix For: 1.0.0
>
> Attachments: NUTCH-516.patch
>
>
> We can not crawl some page due to a robots restriction. In this case we update the db with the Metada: _pst_:robots_denied(18) , we add the status code 3 and we change the fecth interval to 67.5 days.
> Unfortunetely the Fetch time is never change, so it keeps generating this page and fetching it every time.
> We should update the schedule fetch in crawldb to reflect to the fetch interval.
> We should add in crawldbreducer:
> case CrawlDatum.STATUS_FETCH_GONE: // permanent failure
> if (old != null)
> result.setSignature(old.getSignature()); // use old signature
> result.setStatus(CrawlDatum.STATUS_DB_GONE);
> result = schedule.setPageGoneSchedule((Text)key, result, prevFetchTime,
> prevModifiedTime, fetch.getFetchTime());
> // set the schedule
> result = schedule.setFetchSchedule((Text)key, result, prevFetchTime,
> prevModifiedTime, fetch.getFetchTime(), fetch.getModifiedTime(), modified);
> break;
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (NUTCH-516) Next fetch time is not set when it is a
CrawlDatum.STATUS_FETCH_GONE
Posted by "Doğacan Güney (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doğacan Güney updated NUTCH-516:
--------------------------------
Comment: was deleted
> Next fetch time is not set when it is a CrawlDatum.STATUS_FETCH_GONE
> --------------------------------------------------------------------
>
> Key: NUTCH-516
> URL: https://issues.apache.org/jira/browse/NUTCH-516
> Project: Nutch
> Issue Type: Bug
> Components: fetcher
> Environment: Java 1.6, Linux 2.6
> Reporter: Emmanuel Joke
> Fix For: 1.0.0
>
>
> We can not crawl some page due to a robots restriction. In this case we update the db with the Metada: _pst_:robots_denied(18) , we add the status code 3 and we change the fecth interval to 67.5 days.
> Unfortunetely the Fetch time is never change, so it keeps generating this page and fetching it every time.
> We should update the schedule fetch in crawldb to reflect to the fetch interval.
> We should add in crawldbreducer:
> case CrawlDatum.STATUS_FETCH_GONE: // permanent failure
> if (old != null)
> result.setSignature(old.getSignature()); // use old signature
> result.setStatus(CrawlDatum.STATUS_DB_GONE);
> result = schedule.setPageGoneSchedule((Text)key, result, prevFetchTime,
> prevModifiedTime, fetch.getFetchTime());
> // set the schedule
> result = schedule.setFetchSchedule((Text)key, result, prevFetchTime,
> prevModifiedTime, fetch.getFetchTime(), fetch.getModifiedTime(), modified);
> break;
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (NUTCH-516) Next fetch time is not set when it is a
CrawlDatum.STATUS_FETCH_GONE
Posted by "Emmanuel Joke (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Emmanuel Joke updated NUTCH-516:
--------------------------------
Attachment: NUTCH-516.patch
I fxied the issue by changing the FetchTime in AbstractFetchSchedule.setPageGoneSchedule.
I also noticed that the FetchTime was not correctly set. It was always 25 days later and not the 30 as defined in nutch-default.xml. Actually the function: Math.round(datum.getFetchInterval() * 1000.0f) always returned the same value "2147483647" which is Interger.MAX_VALUE. It didn't convert the value in LONG but in INT. So in order to force it to convert in LONG I changed 1000.0f by 1000.0d.
Now the fecthtime will set be correctly according to the FetchInterval.
> Next fetch time is not set when it is a CrawlDatum.STATUS_FETCH_GONE
> --------------------------------------------------------------------
>
> Key: NUTCH-516
> URL: https://issues.apache.org/jira/browse/NUTCH-516
> Project: Nutch
> Issue Type: Bug
> Components: fetcher
> Environment: Java 1.6, Linux 2.6
> Reporter: Emmanuel Joke
> Fix For: 1.0.0
>
> Attachments: NUTCH-516.patch
>
>
> We can not crawl some page due to a robots restriction. In this case we update the db with the Metada: _pst_:robots_denied(18) , we add the status code 3 and we change the fecth interval to 67.5 days.
> Unfortunetely the Fetch time is never change, so it keeps generating this page and fetching it every time.
> We should update the schedule fetch in crawldb to reflect to the fetch interval.
> We should add in crawldbreducer:
> case CrawlDatum.STATUS_FETCH_GONE: // permanent failure
> if (old != null)
> result.setSignature(old.getSignature()); // use old signature
> result.setStatus(CrawlDatum.STATUS_DB_GONE);
> result = schedule.setPageGoneSchedule((Text)key, result, prevFetchTime,
> prevModifiedTime, fetch.getFetchTime());
> // set the schedule
> result = schedule.setFetchSchedule((Text)key, result, prevFetchTime,
> prevModifiedTime, fetch.getFetchTime(), fetch.getModifiedTime(), modified);
> break;
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (NUTCH-516) Next fetch time is not set when it is
a CrawlDatum.STATUS_FETCH_GONE
Posted by "Doğacan Güney (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12513249 ]
Doğacan Güney commented on NUTCH-516:
-------------------------------------
Actually schedule.setPageGoneSchedule updates the next fetch time. The problem is that default fetchInterval is tiny (30 seconds) so setting next fetch time to fetchInterval * 1.5 (as setPageGoneSchedule does) pushes next fetch time only 45 seconds later. I think if you retry with NUTCH-515 your problem will be solved. Note: After applying NUTCH-515, you have to re-inject and run parse and updatedb-s for all segmens again (but you don't need to refetch).
> Next fetch time is not set when it is a CrawlDatum.STATUS_FETCH_GONE
> --------------------------------------------------------------------
>
> Key: NUTCH-516
> URL: https://issues.apache.org/jira/browse/NUTCH-516
> Project: Nutch
> Issue Type: Bug
> Components: fetcher
> Environment: Java 1.6, Linux 2.6
> Reporter: Emmanuel Joke
> Fix For: 1.0.0
>
>
> We can not crawl some page due to a robots restriction. In this case we update the db with the Metada: _pst_:robots_denied(18) , we add the status code 3 and we change the fecth interval to 67.5 days.
> Unfortunetely the Fetch time is never change, so it keeps generating this page and fetching it every time.
> We should update the schedule fetch in crawldb to reflect to the fetch interval.
> We should add in crawldbreducer:
> case CrawlDatum.STATUS_FETCH_GONE: // permanent failure
> if (old != null)
> result.setSignature(old.getSignature()); // use old signature
> result.setStatus(CrawlDatum.STATUS_DB_GONE);
> result = schedule.setPageGoneSchedule((Text)key, result, prevFetchTime,
> prevModifiedTime, fetch.getFetchTime());
> // set the schedule
> result = schedule.setFetchSchedule((Text)key, result, prevFetchTime,
> prevModifiedTime, fetch.getFetchTime(), fetch.getModifiedTime(), modified);
> break;
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (NUTCH-516) Next fetch time is not set when it is
a CrawlDatum.STATUS_FETCH_GONE
Posted by "Doğacan Güney (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12513255 ]
Doğacan Güney commented on NUTCH-516:
-------------------------------------
My last (deleted) comment is completely wrong. You are right that fetch time is not updated.
I think it would be better to update setPageGoneSchedule to set next fetch time instead of calling setFetchSchedule in CrawlDbReducer.
Can you attach your change as a patch?
> Next fetch time is not set when it is a CrawlDatum.STATUS_FETCH_GONE
> --------------------------------------------------------------------
>
> Key: NUTCH-516
> URL: https://issues.apache.org/jira/browse/NUTCH-516
> Project: Nutch
> Issue Type: Bug
> Components: fetcher
> Environment: Java 1.6, Linux 2.6
> Reporter: Emmanuel Joke
> Fix For: 1.0.0
>
>
> We can not crawl some page due to a robots restriction. In this case we update the db with the Metada: _pst_:robots_denied(18) , we add the status code 3 and we change the fecth interval to 67.5 days.
> Unfortunetely the Fetch time is never change, so it keeps generating this page and fetching it every time.
> We should update the schedule fetch in crawldb to reflect to the fetch interval.
> We should add in crawldbreducer:
> case CrawlDatum.STATUS_FETCH_GONE: // permanent failure
> if (old != null)
> result.setSignature(old.getSignature()); // use old signature
> result.setStatus(CrawlDatum.STATUS_DB_GONE);
> result = schedule.setPageGoneSchedule((Text)key, result, prevFetchTime,
> prevModifiedTime, fetch.getFetchTime());
> // set the schedule
> result = schedule.setFetchSchedule((Text)key, result, prevFetchTime,
> prevModifiedTime, fetch.getFetchTime(), fetch.getModifiedTime(), modified);
> break;
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (NUTCH-516) Next fetch time is not set when it is
a CrawlDatum.STATUS_FETCH_GONE
Posted by "Doğacan Güney (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doğacan Güney resolved NUTCH-516.
---------------------------------
Resolution: Fixed
Patch committed in rev. 559742.
> Next fetch time is not set when it is a CrawlDatum.STATUS_FETCH_GONE
> --------------------------------------------------------------------
>
> Key: NUTCH-516
> URL: https://issues.apache.org/jira/browse/NUTCH-516
> Project: Nutch
> Issue Type: Bug
> Components: fetcher
> Environment: Java 1.6, Linux 2.6
> Reporter: Emmanuel Joke
> Fix For: 1.0.0
>
> Attachments: NUTCH-516.patch
>
>
> We can not crawl some page due to a robots restriction. In this case we update the db with the Metada: _pst_:robots_denied(18) , we add the status code 3 and we change the fecth interval to 67.5 days.
> Unfortunetely the Fetch time is never change, so it keeps generating this page and fetching it every time.
> We should update the schedule fetch in crawldb to reflect to the fetch interval.
> We should add in crawldbreducer:
> case CrawlDatum.STATUS_FETCH_GONE: // permanent failure
> if (old != null)
> result.setSignature(old.getSignature()); // use old signature
> result.setStatus(CrawlDatum.STATUS_DB_GONE);
> result = schedule.setPageGoneSchedule((Text)key, result, prevFetchTime,
> prevModifiedTime, fetch.getFetchTime());
> // set the schedule
> result = schedule.setFetchSchedule((Text)key, result, prevFetchTime,
> prevModifiedTime, fetch.getFetchTime(), fetch.getModifiedTime(), modified);
> break;
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (NUTCH-516) Next fetch time is not set when it is
a CrawlDatum.STATUS_FETCH_GONE
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12515954 ]
Hudson commented on NUTCH-516:
------------------------------
Integrated in Nutch-Nightly #162 (See [http://lucene.zones.apache.org:8080/hudson/job/Nutch-Nightly/162/])
> Next fetch time is not set when it is a CrawlDatum.STATUS_FETCH_GONE
> --------------------------------------------------------------------
>
> Key: NUTCH-516
> URL: https://issues.apache.org/jira/browse/NUTCH-516
> Project: Nutch
> Issue Type: Bug
> Components: fetcher
> Environment: Java 1.6, Linux 2.6
> Reporter: Emmanuel Joke
> Fix For: 1.0.0
>
> Attachments: NUTCH-516.patch
>
>
> We can not crawl some page due to a robots restriction. In this case we update the db with the Metada: _pst_:robots_denied(18) , we add the status code 3 and we change the fecth interval to 67.5 days.
> Unfortunetely the Fetch time is never change, so it keeps generating this page and fetching it every time.
> We should update the schedule fetch in crawldb to reflect to the fetch interval.
> We should add in crawldbreducer:
> case CrawlDatum.STATUS_FETCH_GONE: // permanent failure
> if (old != null)
> result.setSignature(old.getSignature()); // use old signature
> result.setStatus(CrawlDatum.STATUS_DB_GONE);
> result = schedule.setPageGoneSchedule((Text)key, result, prevFetchTime,
> prevModifiedTime, fetch.getFetchTime());
> // set the schedule
> result = schedule.setFetchSchedule((Text)key, result, prevFetchTime,
> prevModifiedTime, fetch.getFetchTime(), fetch.getModifiedTime(), modified);
> break;
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Closed: (NUTCH-516) Next fetch time is not set when it is a
CrawlDatum.STATUS_FETCH_GONE
Posted by "Doğacan Güney (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doğacan Güney closed NUTCH-516.
-------------------------------
Resolved and committed.
> Next fetch time is not set when it is a CrawlDatum.STATUS_FETCH_GONE
> --------------------------------------------------------------------
>
> Key: NUTCH-516
> URL: https://issues.apache.org/jira/browse/NUTCH-516
> Project: Nutch
> Issue Type: Bug
> Components: fetcher
> Environment: Java 1.6, Linux 2.6
> Reporter: Emmanuel Joke
> Fix For: 1.0.0
>
> Attachments: NUTCH-516.patch
>
>
> We can not crawl some page due to a robots restriction. In this case we update the db with the Metada: _pst_:robots_denied(18) , we add the status code 3 and we change the fecth interval to 67.5 days.
> Unfortunetely the Fetch time is never change, so it keeps generating this page and fetching it every time.
> We should update the schedule fetch in crawldb to reflect to the fetch interval.
> We should add in crawldbreducer:
> case CrawlDatum.STATUS_FETCH_GONE: // permanent failure
> if (old != null)
> result.setSignature(old.getSignature()); // use old signature
> result.setStatus(CrawlDatum.STATUS_DB_GONE);
> result = schedule.setPageGoneSchedule((Text)key, result, prevFetchTime,
> prevModifiedTime, fetch.getFetchTime());
> // set the schedule
> result = schedule.setFetchSchedule((Text)key, result, prevFetchTime,
> prevModifiedTime, fetch.getFetchTime(), fetch.getModifiedTime(), modified);
> break;
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (NUTCH-516) Next fetch time is not set when it is
a CrawlDatum.STATUS_FETCH_GONE
Posted by "Andrzej Bialecki (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12513258 ]
Andrzej Bialecki commented on NUTCH-516:
-----------------------------------------
setPageGoneSchedule method was specifically added to handle this case. The bug is in this method (in the Schedule implementation) and it should be fixed there, instead of calling setFetchSchedule.
> Next fetch time is not set when it is a CrawlDatum.STATUS_FETCH_GONE
> --------------------------------------------------------------------
>
> Key: NUTCH-516
> URL: https://issues.apache.org/jira/browse/NUTCH-516
> Project: Nutch
> Issue Type: Bug
> Components: fetcher
> Environment: Java 1.6, Linux 2.6
> Reporter: Emmanuel Joke
> Fix For: 1.0.0
>
>
> We can not crawl some page due to a robots restriction. In this case we update the db with the Metada: _pst_:robots_denied(18) , we add the status code 3 and we change the fecth interval to 67.5 days.
> Unfortunetely the Fetch time is never change, so it keeps generating this page and fetching it every time.
> We should update the schedule fetch in crawldb to reflect to the fetch interval.
> We should add in crawldbreducer:
> case CrawlDatum.STATUS_FETCH_GONE: // permanent failure
> if (old != null)
> result.setSignature(old.getSignature()); // use old signature
> result.setStatus(CrawlDatum.STATUS_DB_GONE);
> result = schedule.setPageGoneSchedule((Text)key, result, prevFetchTime,
> prevModifiedTime, fetch.getFetchTime());
> // set the schedule
> result = schedule.setFetchSchedule((Text)key, result, prevFetchTime,
> prevModifiedTime, fetch.getFetchTime(), fetch.getModifiedTime(), modified);
> break;
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.