[jira] [Assigned] (CONNECTORS-854) Enable STALE_CONNECTION_CHECK

["Karl Wright (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/CONNECTORS-854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Karl Wright reassigned CONNECTORS-854:
--------------------------------------

    Assignee: Karl Wright

> Enable STALE_CONNECTION_CHECK
> -----------------------------
>
>                 Key: CONNECTORS-854
>                 URL: https://issues.apache.org/jira/browse/CONNECTORS-854
>             Project: ManifoldCF
>          Issue Type: Improvement
>          Components: Web connector
>    Affects Versions: ManifoldCF 1.4.1
>            Reporter: Shinichiro Abe
>            Assignee: Karl Wright
>            Priority: Minor
>             Fix For: ManifoldCF 1.5
>
>
> When crawling some sites( < 1000 docs), sometimes manifoldcf.log shows the following "The target server failed to respond" messages. It seems that NoHttpResponseException is thrown at ThrottledFetcher.
> {noformat}
>  WARN 2014-01-09 12:39:16,701 (Worker thread '10') - Pre-ingest service interruption reported for job 1389238470356 connection '1': Timed out waiting for response for 'http://www.rondhuit.com/?p=1890': The target server failed to respond
>  WARN 2014-01-09 12:39:55,509 (Worker thread '7') - Pre-ingest service interruption reported for job 1389238470356 connection '1': Timed out waiting for response for 'http://www.rondhuit.com/?p=675': The target server failed to respond
> {noformat}
> The fetching that page after retry time(15 minutes) passed was running successfully.
> I tried to change a httpclient configuration then I confirmed that massage was not shown.
> {noformat}
> +++ connectors/webcrawler/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/webcrawler/ThrottledFetcher.java
> @@ -463,7 +463,7 @@
>          BasicHttpParams params = new BasicHttpParams();
>          params.setParameter(ClientPNames.DEFAULT_HOST,fetchHost);
>          params.setBooleanParameter(CoreConnectionPNames.TCP_NODELAY,true);
> -        params.setBooleanParameter(CoreConnectionPNames.STALE_CONNECTION_CHECK,false);
> +        params.setBooleanParameter(CoreConnectionPNames.STALE_CONNECTION_CHECK,true);
>          params.setBooleanParameter(ClientPNames.ALLOW_CIRCULAR_REDIRECTS,true);
> {noformat}
> I know two users who are hitting this issue and have resolved it by turning on stale connection check.
> The crawling job is done more quickly than the check is false because there are not retry fetches.
> May I switch false to true in stale connection check as well as SolrConnector's httpclient configuration?



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)