You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@nutch.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2022/01/11 14:19:00 UTC

[jira] [Commented] (NUTCH-2929) Fetcher: start threads slowly to avoid that resources are temporarily exhausted

    [ https://issues.apache.org/jira/browse/NUTCH-2929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17472762#comment-17472762 ] 

ASF GitHub Bot commented on NUTCH-2929:
---------------------------------------

sebastian-nagel opened a new pull request #722:
URL: https://github.com/apache/nutch/pull/722


   Sleep for a configurable delay (fetcher.threads.start.delay) before starting the next Fetcher thread to avoid that resources (DNS, Tika XML parser pools) are temporarily exhausted when Fetcher threads fetch the first pages simultaneously
   
   The default delay (10 milliseconds) practically does not impact the behavior/performance of Fetcher - 100 threads are started per second. By choosing a longer delay (500 milliseconds) all Tika warnings about the SAXParser pool size disappeared.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@nutch.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


> Fetcher: start threads slowly to avoid that resources are temporarily exhausted
> -------------------------------------------------------------------------------
>
>                 Key: NUTCH-2929
>                 URL: https://issues.apache.org/jira/browse/NUTCH-2929
>             Project: Nutch
>          Issue Type: Improvement
>    Affects Versions: 1.18
>            Reporter: Sebastian Nagel
>            Priority: Minor
>             Fix For: 1.19
>
>
> Fetcher spins all threads without any delay. This may cause that certain resources are temporarily exhausted if all threads start fetching the first pages simultaneously.
> The issue has been observed by Tika warnings about overuse of the SAXParser pool which appeared only during the first 2-5 minutes of fetching a segment. See https://lists.apache.org/thread/lo6b9wdlxy2lz12wmosldgl9x9ov1cks - adding a short delay between thread launches makes the warnings disappear.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)