[jira] [Created] (NUTCH-2653) ProtocolFactory.getProtocol(url) creates separate plugin instances for http/https

["Sebastian Nagel (JIRA)" <ji...@apache.org>]

Sebastian Nagel created NUTCH-2653:
--------------------------------------

             Summary: ProtocolFactory.getProtocol(url) creates separate plugin instances for http/https
                 Key: NUTCH-2653
                 URL: https://issues.apache.org/jira/browse/NUTCH-2653
             Project: Nutch
          Issue Type: Improvement
          Components: fetcher, protocol
    Affects Versions: 1.15
            Reporter: Sebastian Nagel
             Fix For: 1.16


Fetcher creates two instances of the protocol-okhttp plugin, one to handle http requests, another for https. The plugin properties are logged during plugin instantiation when calling {{setConf(...)}}:
{noformat}
2018-10-11 13:28:34,417 INFO [FetcherThread] org.apache.nutch.fetcher.FetcherThread: FetcherThread 40 fetching http://...
...
2018-10-11 13:28:35,099 INFO [FetcherThread] org.apache.nutch.protocol.okhttp.OkHttp: http.proxy.host = null
2018-10-11 13:28:35,100 INFO [FetcherThread] org.apache.nutch.protocol.okhttp.OkHttp: http.proxy.port = 8080
...
2018-10-11 13:28:36,864 INFO [FetcherThread] org.apache.nutch.fetcher.FetcherThread: FetcherThread 87 fetching https://...
...
2018-10-11 13:28:36,864 INFO [FetcherThread] org.apache.nutch.protocol.okhttp.OkHttp: http.proxy.host = null
2018-10-11 13:28:36,864 INFO [FetcherThread] org.apache.nutch.protocol.okhttp.OkHttp: http.proxy.port = 8080
{noformat}

The question is whether this is the correct behavior for plugins supporting multiple protocols (http and https)? It may cause that connection pooling and other network optimizations do not work as expected. Of course, it's correct if different plugins are required, e.g., for ftp or the local file system.

(seen while reviewing the behavior of fetcher with fix for NUTCH-2625 applied)




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)