You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by "Markus Jelsma (JIRA)" <ji...@apache.org> on 2011/07/16 19:31:59 UTC
[jira] [Created] (NUTCH-1057) Make fetcher thread time out
configurable
Make fetcher thread time out configurable
-----------------------------------------
Key: NUTCH-1057
URL: https://issues.apache.org/jira/browse/NUTCH-1057
Project: Nutch
Issue Type: Improvement
Components: fetcher
Reporter: Markus Jelsma
Assignee: Markus Jelsma
Fix For: 1.4, 2.0
The fetcher sets a time out value based of half the mapred.task.timeout value. This is not a proper value for all cases. Add an option (fetcher.thread.timeout.divisor) to configure the divisor used and default it to two.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (NUTCH-1057) Make fetcher thread time out
configurable
Posted by "Markus Jelsma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Markus Jelsma updated NUTCH-1057:
---------------------------------
Fix Version/s: (was: 2.0)
> Make fetcher thread time out configurable
> -----------------------------------------
>
> Key: NUTCH-1057
> URL: https://issues.apache.org/jira/browse/NUTCH-1057
> Project: Nutch
> Issue Type: Improvement
> Components: fetcher
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.4
>
> Attachments: NUTCH-1057-1.4-1.patch
>
>
> The fetcher sets a time out value based of half the mapred.task.timeout value. This is not a proper value for all cases. Add an option (fetcher.thread.timeout.divisor) to configure the divisor used and default it to two.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1057) Make fetcher thread time out
configurable
Posted by "Julien Nioche (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13090081#comment-13090081 ]
Julien Nioche commented on NUTCH-1057:
--------------------------------------
Haven't you committed it already? Or do you mean for trunk?
> Make fetcher thread time out configurable
> -----------------------------------------
>
> Key: NUTCH-1057
> URL: https://issues.apache.org/jira/browse/NUTCH-1057
> Project: Nutch
> Issue Type: Improvement
> Components: fetcher
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.4, 2.0
>
> Attachments: NUTCH-1057-1.4-1.patch
>
>
> The fetcher sets a time out value based of half the mapred.task.timeout value. This is not a proper value for all cases. Add an option (fetcher.thread.timeout.divisor) to configure the divisor used and default it to two.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (NUTCH-1057) Make fetcher thread time out
configurable
Posted by "Markus Jelsma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Markus Jelsma updated NUTCH-1057:
---------------------------------
Attachment: NUTCH-1057-1.4-1.patch
Patch for 1.4. There's also a diff for NUTCH-1037 in the config file which hasn't been committed yet.
> Make fetcher thread time out configurable
> -----------------------------------------
>
> Key: NUTCH-1057
> URL: https://issues.apache.org/jira/browse/NUTCH-1057
> Project: Nutch
> Issue Type: Improvement
> Components: fetcher
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.4, 2.0
>
> Attachments: NUTCH-1057-1.4-1.patch
>
>
> The fetcher sets a time out value based of half the mapred.task.timeout value. This is not a proper value for all cases. Add an option (fetcher.thread.timeout.divisor) to configure the divisor used and default it to two.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (NUTCH-1057) Make fetcher thread time out
configurable
Posted by "Markus Jelsma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Markus Jelsma resolved NUTCH-1057.
----------------------------------
Resolution: Fixed
Resolved for 1.4, see NUTCH-1104 for 2.0
> Make fetcher thread time out configurable
> -----------------------------------------
>
> Key: NUTCH-1057
> URL: https://issues.apache.org/jira/browse/NUTCH-1057
> Project: Nutch
> Issue Type: Improvement
> Components: fetcher
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.4
>
> Attachments: NUTCH-1057-1.4-1.patch
>
>
> The fetcher sets a time out value based of half the mapred.task.timeout value. This is not a proper value for all cases. Add an option (fetcher.thread.timeout.divisor) to configure the divisor used and default it to two.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1057) Make fetcher thread time out
configurable
Posted by "Markus Jelsma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13090106#comment-13090106 ]
Markus Jelsma commented on NUTCH-1057:
--------------------------------------
Sorry, wrong issue!
> Make fetcher thread time out configurable
> -----------------------------------------
>
> Key: NUTCH-1057
> URL: https://issues.apache.org/jira/browse/NUTCH-1057
> Project: Nutch
> Issue Type: Improvement
> Components: fetcher
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.4, 2.0
>
> Attachments: NUTCH-1057-1.4-1.patch
>
>
> The fetcher sets a time out value based of half the mapred.task.timeout value. This is not a proper value for all cases. Add an option (fetcher.thread.timeout.divisor) to configure the divisor used and default it to two.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1057) Make fetcher thread time out
configurable
Posted by "Julien Nioche (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13067678#comment-13067678 ]
Julien Nioche commented on NUTCH-1057:
--------------------------------------
Apart from the part related to NUTCH-1037 which will need removing unless it is committed your patch looks OK.
Just to make sure I understand what the issue is : a hadoop fetch task can fail because the timeout for fetch threads is too long. Is that right?
> Make fetcher thread time out configurable
> -----------------------------------------
>
> Key: NUTCH-1057
> URL: https://issues.apache.org/jira/browse/NUTCH-1057
> Project: Nutch
> Issue Type: Improvement
> Components: fetcher
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.4, 2.0
>
> Attachments: NUTCH-1057-1.4-1.patch
>
>
> The fetcher sets a time out value based of half the mapred.task.timeout value. This is not a proper value for all cases. Add an option (fetcher.thread.timeout.divisor) to configure the divisor used and default it to two.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1057) Make fetcher thread time out
configurable
Posted by "Markus Jelsma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13067665#comment-13067665 ]
Markus Jelsma commented on NUTCH-1057:
--------------------------------------
Any comments or objections? Any better methods?
> Make fetcher thread time out configurable
> -----------------------------------------
>
> Key: NUTCH-1057
> URL: https://issues.apache.org/jira/browse/NUTCH-1057
> Project: Nutch
> Issue Type: Improvement
> Components: fetcher
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.4, 2.0
>
> Attachments: NUTCH-1057-1.4-1.patch
>
>
> The fetcher sets a time out value based of half the mapred.task.timeout value. This is not a proper value for all cases. Add an option (fetcher.thread.timeout.divisor) to configure the divisor used and default it to two.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1057) Make fetcher thread time out
configurable
Posted by "Markus Jelsma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13089762#comment-13089762 ]
Markus Jelsma commented on NUTCH-1057:
--------------------------------------
I'd like to commit this issue this friday unless there are objections or other comments.
> Make fetcher thread time out configurable
> -----------------------------------------
>
> Key: NUTCH-1057
> URL: https://issues.apache.org/jira/browse/NUTCH-1057
> Project: Nutch
> Issue Type: Improvement
> Components: fetcher
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.4, 2.0
>
> Attachments: NUTCH-1057-1.4-1.patch
>
>
> The fetcher sets a time out value based of half the mapred.task.timeout value. This is not a proper value for all cases. Add an option (fetcher.thread.timeout.divisor) to configure the divisor used and default it to two.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (NUTCH-1057) Make fetcher thread time out
configurable
Posted by "Markus Jelsma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Markus Jelsma updated NUTCH-1057:
---------------------------------
Patch Info: [Patch Available]
> Make fetcher thread time out configurable
> -----------------------------------------
>
> Key: NUTCH-1057
> URL: https://issues.apache.org/jira/browse/NUTCH-1057
> Project: Nutch
> Issue Type: Improvement
> Components: fetcher
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.4, 2.0
>
> Attachments: NUTCH-1057-1.4-1.patch
>
>
> The fetcher sets a time out value based of half the mapred.task.timeout value. This is not a proper value for all cases. Add an option (fetcher.thread.timeout.divisor) to configure the divisor used and default it to two.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (NUTCH-1057) Make fetcher thread time out
configurable
Posted by "Markus Jelsma (Closed) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Markus Jelsma closed NUTCH-1057.
--------------------------------
Bulk close of resolved issues of 1.4. bulkclose-1.4-20111220
> Make fetcher thread time out configurable
> -----------------------------------------
>
> Key: NUTCH-1057
> URL: https://issues.apache.org/jira/browse/NUTCH-1057
> Project: Nutch
> Issue Type: Improvement
> Components: fetcher
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.4
>
> Attachments: NUTCH-1057-1.4-1.patch
>
>
> The fetcher sets a time out value based of half the mapred.task.timeout value. This is not a proper value for all cases. Add an option (fetcher.thread.timeout.divisor) to configure the divisor used and default it to two.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1057) Make fetcher thread time out
configurable
Posted by "Markus Jelsma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13067686#comment-13067686 ]
Markus Jelsma commented on NUTCH-1057:
--------------------------------------
No. This is a tuning option for users that experience very long pauses in the merge phase after a map finishes. It takes long because there are many GB's of map output and/or slow IO.
To prevent the task tracker from killing the merge (default 600s time out) users need to raise the mapred.timeout value to a value higher than the actual duration of the merge phase.
Fetcher threads have a time out that is configured to be half the tasktracker time out value. This means that with a high (e.g. 20m) task timeout, the fetcher will wait 10m before killing hanging threads. This is a waste of time. In large crawl there are always a few threads unable to finish properly. Killing them sooner makes the merge begin earlier.
Sorry if i was unclear before.
> Make fetcher thread time out configurable
> -----------------------------------------
>
> Key: NUTCH-1057
> URL: https://issues.apache.org/jira/browse/NUTCH-1057
> Project: Nutch
> Issue Type: Improvement
> Components: fetcher
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.4, 2.0
>
> Attachments: NUTCH-1057-1.4-1.patch
>
>
> The fetcher sets a time out value based of half the mapred.task.timeout value. This is not a proper value for all cases. Add an option (fetcher.thread.timeout.divisor) to configure the divisor used and default it to two.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1057) Make fetcher thread time out
configurable
Posted by "Markus Jelsma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13067791#comment-13067791 ]
Markus Jelsma commented on NUTCH-1057:
--------------------------------------
Committed for 1.4 in rev. 1148406.
> Make fetcher thread time out configurable
> -----------------------------------------
>
> Key: NUTCH-1057
> URL: https://issues.apache.org/jira/browse/NUTCH-1057
> Project: Nutch
> Issue Type: Improvement
> Components: fetcher
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.4, 2.0
>
> Attachments: NUTCH-1057-1.4-1.patch
>
>
> The fetcher sets a time out value based of half the mapred.task.timeout value. This is not a proper value for all cases. Add an option (fetcher.thread.timeout.divisor) to configure the divisor used and default it to two.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira