You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by "Andrzej Bialecki (JIRA)" <ji...@apache.org> on 2005/06/05 22:44:39 UTC
[jira] Created: (NUTCH-61) Adaptive re-fetch interval. Detecting umodified content
Adaptive re-fetch interval. Detecting umodified content
-------------------------------------------------------
Key: NUTCH-61
URL: http://issues.apache.org/jira/browse/NUTCH-61
Project: Nutch
Type: New Feature
Components: fetcher
Reporter: Andrzej Bialecki
Assigned to: Andrzej Bialecki
Currently Nutch doesn't adjust automatically its re-fetch period, no matter if individual pages change seldom or frequently. The goal of these changes is to extend the current codebase to support various possible adjustments to re-fetch times and intervals, and specifically a re-fetch schedule which tries to adapt the period between consecutive fetches to the period of content changes.
Also, these patches implement checking if the content has changed since last fetching; protocol plugins are also changed to make use of this information, so that if content is unmodified it doesn't have to be fetched and processed.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
Re: [jira] Created: (NUTCH-61) Adaptive re-fetch interval. Detecting
umodified content
Posted by Andrzej Bialecki <ab...@getopt.org>.
yoursoft@freemail.hu wrote:
> Dear Andrzej,
>
> This is very interesting patch, but I have a question:
> - If page isn't modified, you don't refetch page.
Correct - that's the whole point of this patch.
> - If you don't refrech page, there is in old segments?
Yes, the page should be in some old segment. This question brings an
interesting dilemma - should I add an option to forcefully refetch a
page, in case you lost the old segment data? Hmmm... In the current code
there is an option "-adddays", but with adjustable interval this doesn't
make much sense.
> - If it's in old segments, the segments data will be increasses, how to
> analize that which segment is deletable?
Well, there is no good answer to that even with the current code... You
can use mergesegs tool to keep only the latest versions of pages. But I
agree, this patch make the problem of handling old segments more serious
- how to "phase out" older segments.
> - If it's in the old segments, the useable index will be larger and
> larger. Because there is a limitation: optimal 2Kbyte RAM / page -> this
> will decrease performance or increasse RAM usage?
The index (Lucene index) will not be larger - the deduplication process
takes care of that. Only the latest version of the content will show up
in the index, and for identical content only the one reachable via the
shortest URL.
>
> Sorry my performance question, this patch is very interesting and usable.
Thanks for good, thought-provoking questions!
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
Re: [jira] Created: (NUTCH-61) Adaptive re-fetch interval. Detecting
umodified content
Posted by "yoursoft@freemail.hu" <yo...@freemail.hu>.
Dear Andrzej,
This is very interesting patch, but I have a question:
- If page isn't modified, you don't refetch page.
- If you don't refrech page, there is in old segments?
- If it's in old segments, the segments data will be increasses, how to
analize that which segment is deletable?
- If it's in the old segments, the useable index will be larger and
larger. Because there is a limitation: optimal 2Kbyte RAM / page -> this
will decrease performance or increasse RAM usage?
Sorry my performance question, this patch is very interesting and usable.
Thanks for your your answer,
Ferenc
Andrzej Bialecki (JIRA) wrotte:
>Adaptive re-fetch interval. Detecting umodified content
>-------------------------------------------------------
>
> Key: NUTCH-61
> URL: http://issues.apache.org/jira/browse/NUTCH-61
> Project: Nutch
> Type: New Feature
> Components: fetcher
> Reporter: Andrzej Bialecki
> Assigned to: Andrzej Bialecki
>
>
>Currently Nutch doesn't adjust automatically its re-fetch period, no matter if individual pages change seldom or frequently. The goal of these changes is to extend the current codebase to support various possible adjustments to re-fetch times and intervals, and specifically a re-fetch schedule which tries to adapt the period between consecutive fetches to the period of content changes.
>
>Also, these patches implement checking if the content has changed since last fetching; protocol plugins are also changed to make use of this information, so that if content is unmodified it doesn't have to be fetched and processed.
>
>
>
Re: [jira] Updated: (NUTCH-61) Adaptive re-fetch interval. Detecting
umodified content
Posted by Andrzej Bialecki <ab...@getopt.org>.
Andrzej Bialecki (JIRA) wrote:
> * improve status reporting throughout all plugins.
Please note, that this is an incompatble change between the
ProtocolStatus implemented patchset in NUTCH-54 and here, so if you
created some segments in between, you will need to refetch them.
I didn't feel that the last version of ProtocolStatus was in such a long
use (3 days?) to merit changing it into a VersionedWritable (and adding
another byte to its size).
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
[jira] Updated: (NUTCH-61) Adaptive re-fetch interval. Detecting umodified content
Posted by "Andrzej Bialecki (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/NUTCH-61?page=all ]
Andrzej Bialecki updated NUTCH-61:
-----------------------------------
Attachment: 20050606.diff
The first round:
* change Page to use a 1-byte float, representing fetchInterval in seconds.
* implement a pluggable FetchSchedule, which adjusts fetchInterval and nextFetchTime
* change FetchListTool and UpdateDatabaseTool to use them. NOTE: it appears there was a bug in FetchListTool, where the fetchlist entries recorded in segments would have their fetchTime increased by 1 week. This is not needed, only pages in WebDB need this.
* improve status reporting throughout all plugins.
* change plugins to detect if the content is unchanged. If possible, plugins will not fetch such content, but in any case they will set their status accordingly.
> Adaptive re-fetch interval. Detecting umodified content
> -------------------------------------------------------
>
> Key: NUTCH-61
> URL: http://issues.apache.org/jira/browse/NUTCH-61
> Project: Nutch
> Type: New Feature
> Components: fetcher
> Reporter: Andrzej Bialecki
> Assignee: Andrzej Bialecki
> Attachments: 20050606.diff
>
> Currently Nutch doesn't adjust automatically its re-fetch period, no matter if individual pages change seldom or frequently. The goal of these changes is to extend the current codebase to support various possible adjustments to re-fetch times and intervals, and specifically a re-fetch schedule which tries to adapt the period between consecutive fetches to the period of content changes.
> Also, these patches implement checking if the content has changed since last fetching; protocol plugins are also changed to make use of this information, so that if content is unmodified it doesn't have to be fetched and processed.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (NUTCH-61) Adaptive re-fetch interval. Detecting umodified content
Posted by "raghavendra prabhu (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/NUTCH-61?page=comments#action_12361131 ]
raghavendra prabhu commented on NUTCH-61:
-----------------------------------------
Will the same thing work for a filesystem
For a file system , We can directly get the modified date store it in the db
The plugins will have a look at the content date and if it is different they will index it
Otherwise they will not fetch it
This can be a solution for file based content
(The thing is it does away entirely with fetch interval and takes decision only based upon file modification date)
> Adaptive re-fetch interval. Detecting umodified content
> -------------------------------------------------------
>
> Key: NUTCH-61
> URL: http://issues.apache.org/jira/browse/NUTCH-61
> Project: Nutch
> Type: New Feature
> Components: fetcher
> Reporter: Andrzej Bialecki
> Assignee: Andrzej Bialecki
> Attachments: 20050606.diff
>
> Currently Nutch doesn't adjust automatically its re-fetch period, no matter if individual pages change seldom or frequently. The goal of these changes is to extend the current codebase to support various possible adjustments to re-fetch times and intervals, and specifically a re-fetch schedule which tries to adapt the period between consecutive fetches to the period of content changes.
> Also, these patches implement checking if the content has changed since last fetching; protocol plugins are also changed to make use of this information, so that if content is unmodified it doesn't have to be fetched and processed.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (NUTCH-61) Adaptive re-fetch interval. Detecting umodified content
Posted by "Andrzej Bialecki (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/NUTCH-61?page=comments#action_12361311 ]
Andrzej Bialecki commented on NUTCH-61:
----------------------------------------
I'm working on this, the patch will be available in a couple of days. I could use then your help with review and testing... ;-)
> Adaptive re-fetch interval. Detecting umodified content
> -------------------------------------------------------
>
> Key: NUTCH-61
> URL: http://issues.apache.org/jira/browse/NUTCH-61
> Project: Nutch
> Type: New Feature
> Components: fetcher
> Reporter: Andrzej Bialecki
> Assignee: Andrzej Bialecki
> Attachments: 20050606.diff
>
> Currently Nutch doesn't adjust automatically its re-fetch period, no matter if individual pages change seldom or frequently. The goal of these changes is to extend the current codebase to support various possible adjustments to re-fetch times and intervals, and specifically a re-fetch schedule which tries to adapt the period between consecutive fetches to the period of content changes.
> Also, these patches implement checking if the content has changed since last fetching; protocol plugins are also changed to make use of this information, so that if content is unmodified it doesn't have to be fetched and processed.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (NUTCH-61) Adaptive re-fetch interval. Detecting umodified content
Posted by "byron miller (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/NUTCH-61?page=comments#action_12361346 ]
byron miller commented on NUTCH-61:
-----------------------------------
Most definately! I'll be happy to give it a whirl!
> Adaptive re-fetch interval. Detecting umodified content
> -------------------------------------------------------
>
> Key: NUTCH-61
> URL: http://issues.apache.org/jira/browse/NUTCH-61
> Project: Nutch
> Type: New Feature
> Components: fetcher
> Reporter: Andrzej Bialecki
> Assignee: Andrzej Bialecki
> Attachments: 20050606.diff
>
> Currently Nutch doesn't adjust automatically its re-fetch period, no matter if individual pages change seldom or frequently. The goal of these changes is to extend the current codebase to support various possible adjustments to re-fetch times and intervals, and specifically a re-fetch schedule which tries to adapt the period between consecutive fetches to the period of content changes.
> Also, these patches implement checking if the content has changed since last fetching; protocol plugins are also changed to make use of this information, so that if content is unmodified it doesn't have to be fetched and processed.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (NUTCH-61) Adaptive re-fetch interval. Detecting umodified content
Posted by "Andrzej Bialecki (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/NUTCH-61?page=comments#action_12361133 ]
Andrzej Bialecki commented on NUTCH-61:
----------------------------------------
This patch already supports this. Anyway, it needs to be significantly re-worked to fit into the current development version.
> Adaptive re-fetch interval. Detecting umodified content
> -------------------------------------------------------
>
> Key: NUTCH-61
> URL: http://issues.apache.org/jira/browse/NUTCH-61
> Project: Nutch
> Type: New Feature
> Components: fetcher
> Reporter: Andrzej Bialecki
> Assignee: Andrzej Bialecki
> Attachments: 20050606.diff
>
> Currently Nutch doesn't adjust automatically its re-fetch period, no matter if individual pages change seldom or frequently. The goal of these changes is to extend the current codebase to support various possible adjustments to re-fetch times and intervals, and specifically a re-fetch schedule which tries to adapt the period between consecutive fetches to the period of content changes.
> Also, these patches implement checking if the content has changed since last fetching; protocol plugins are also changed to make use of this information, so that if content is unmodified it doesn't have to be fetched and processed.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Updated: (NUTCH-61) Adaptive re-fetch interval. Detecting
umodified content
Posted by "Andrzej Bialecki (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/NUTCH-61?page=all ]
Andrzej Bialecki updated NUTCH-61:
-----------------------------------
Attachment: nutch-61-417287.patch
This patch, besides bringing it up-to-date with the trunk/, also adds a maximum cap on fetch interval and a better strategy for merging records in Injector.
> Adaptive re-fetch interval. Detecting umodified content
> -------------------------------------------------------
>
> Key: NUTCH-61
> URL: http://issues.apache.org/jira/browse/NUTCH-61
> Project: Nutch
> Type: New Feature
> Components: fetcher
> Reporter: Andrzej Bialecki
> Assignee: Andrzej Bialecki
> Attachments: 20050606.diff, 20051230.txt, 20060227.txt, nutch-61-417287.patch
>
> Currently Nutch doesn't adjust automatically its re-fetch period, no matter if individual pages change seldom or frequently. The goal of these changes is to extend the current codebase to support various possible adjustments to re-fetch times and intervals, and specifically a re-fetch schedule which tries to adapt the period between consecutive fetches to the period of content changes.
> Also, these patches implement checking if the content has changed since last fetching; protocol plugins are also changed to make use of this information, so that if content is unmodified it doesn't have to be fetched and processed.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (NUTCH-61) Adaptive re-fetch interval. Detecting
umodified content
Posted by "Jerome Charron (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/NUTCH-61?page=comments#action_12368050 ]
Jerome Charron commented on NUTCH-61:
-------------------------------------
Not an objection, but a simple comment.
Why not making FetchSchedule a new ExtensionPoint and then DefaultFetchSchedule and AdaptiveFetchSchedule some fetch schedule plugins?
> Adaptive re-fetch interval. Detecting umodified content
> -------------------------------------------------------
>
> Key: NUTCH-61
> URL: http://issues.apache.org/jira/browse/NUTCH-61
> Project: Nutch
> Type: New Feature
> Components: fetcher
> Reporter: Andrzej Bialecki
> Assignee: Andrzej Bialecki
> Attachments: 20050606.diff, 20051230.txt, 20060227.txt
>
> Currently Nutch doesn't adjust automatically its re-fetch period, no matter if individual pages change seldom or frequently. The goal of these changes is to extend the current codebase to support various possible adjustments to re-fetch times and intervals, and specifically a re-fetch schedule which tries to adapt the period between consecutive fetches to the period of content changes.
> Also, these patches implement checking if the content has changed since last fetching; protocol plugins are also changed to make use of this information, so that if content is unmodified it doesn't have to be fetched and processed.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (NUTCH-61) Adaptive re-fetch interval. Detecting umodified content
Posted by "byron miller (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/NUTCH-61?page=comments#action_12361302 ]
byron miller commented on NUTCH-61:
-----------------------------------
Is there a patch modified for the current branch or should i take a stab at this?
> Adaptive re-fetch interval. Detecting umodified content
> -------------------------------------------------------
>
> Key: NUTCH-61
> URL: http://issues.apache.org/jira/browse/NUTCH-61
> Project: Nutch
> Type: New Feature
> Components: fetcher
> Reporter: Andrzej Bialecki
> Assignee: Andrzej Bialecki
> Attachments: 20050606.diff
>
> Currently Nutch doesn't adjust automatically its re-fetch period, no matter if individual pages change seldom or frequently. The goal of these changes is to extend the current codebase to support various possible adjustments to re-fetch times and intervals, and specifically a re-fetch schedule which tries to adapt the period between consecutive fetches to the period of content changes.
> Also, these patches implement checking if the content has changed since last fetching; protocol plugins are also changed to make use of this information, so that if content is unmodified it doesn't have to be fetched and processed.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (NUTCH-61) Adaptive re-fetch interval. Detecting
umodified content
Posted by "Andrzej Bialecki (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/NUTCH-61?page=comments#action_12368051 ]
Andrzej Bialecki commented on NUTCH-61:
----------------------------------------
I contemplated this for a while, and then decided against it.
The main reason was that currently most of the "pluggable" extensions that result in running a single selected plugin are handled using a simple Factory pattern; as opposed to ChainedFilter pattern, where we use extension points.
I guess the original reason was that implementations would almost always consist of a single class, so it didn't make sense to complicate it and require the whole plugin infrastructure ... It would be the same in this case (just a single class), so I followed the same pattern.
It's easy to change this to use an extension point, if people prefer it this way.
> Adaptive re-fetch interval. Detecting umodified content
> -------------------------------------------------------
>
> Key: NUTCH-61
> URL: http://issues.apache.org/jira/browse/NUTCH-61
> Project: Nutch
> Type: New Feature
> Components: fetcher
> Reporter: Andrzej Bialecki
> Assignee: Andrzej Bialecki
> Attachments: 20050606.diff, 20051230.txt, 20060227.txt
>
> Currently Nutch doesn't adjust automatically its re-fetch period, no matter if individual pages change seldom or frequently. The goal of these changes is to extend the current codebase to support various possible adjustments to re-fetch times and intervals, and specifically a re-fetch schedule which tries to adapt the period between consecutive fetches to the period of content changes.
> Also, these patches implement checking if the content has changed since last fetching; protocol plugins are also changed to make use of this information, so that if content is unmodified it doesn't have to be fetched and processed.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Updated: (NUTCH-61) Adaptive re-fetch interval. Detecting umodified content
Posted by "Andrzej Bialecki (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/NUTCH-61?page=all ]
Andrzej Bialecki updated NUTCH-61:
-----------------------------------
Attachment: 20051230.txt
Updated version for the latest mapred branch.
> Adaptive re-fetch interval. Detecting umodified content
> -------------------------------------------------------
>
> Key: NUTCH-61
> URL: http://issues.apache.org/jira/browse/NUTCH-61
> Project: Nutch
> Type: New Feature
> Components: fetcher
> Reporter: Andrzej Bialecki
> Assignee: Andrzej Bialecki
> Attachments: 20050606.diff, 20051230.txt
>
> Currently Nutch doesn't adjust automatically its re-fetch period, no matter if individual pages change seldom or frequently. The goal of these changes is to extend the current codebase to support various possible adjustments to re-fetch times and intervals, and specifically a re-fetch schedule which tries to adapt the period between consecutive fetches to the period of content changes.
> Also, these patches implement checking if the content has changed since last fetching; protocol plugins are also changed to make use of this information, so that if content is unmodified it doesn't have to be fetched and processed.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Updated: (NUTCH-61) Adaptive re-fetch interval. Detecting
umodified content
Posted by "Andrzej Bialecki (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/NUTCH-61?page=all ]
Andrzej Bialecki updated NUTCH-61:
-----------------------------------
Attachment: 20060227.txt
This patch is updated to the current trunk/ . The default configuration works as before, and uses DefaultFetchSchedule.
If there are no major objections I will commit it shortly.
> Adaptive re-fetch interval. Detecting umodified content
> -------------------------------------------------------
>
> Key: NUTCH-61
> URL: http://issues.apache.org/jira/browse/NUTCH-61
> Project: Nutch
> Type: New Feature
> Components: fetcher
> Reporter: Andrzej Bialecki
> Assignee: Andrzej Bialecki
> Attachments: 20050606.diff, 20051230.txt, 20060227.txt
>
> Currently Nutch doesn't adjust automatically its re-fetch period, no matter if individual pages change seldom or frequently. The goal of these changes is to extend the current codebase to support various possible adjustments to re-fetch times and intervals, and specifically a re-fetch schedule which tries to adapt the period between consecutive fetches to the period of content changes.
> Also, these patches implement checking if the content has changed since last fetching; protocol plugins are also changed to make use of this information, so that if content is unmodified it doesn't have to be fetched and processed.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira