[jira] [Commented] (NUTCH-2576) HTTP protocol plugin based on okhttp

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NUTCH-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16468750#comment-16468750 ] 

ASF GitHub Bot commented on NUTCH-2576:
---------------------------------------

sebastian-nagel opened a new pull request #328: NUTCH-2576 HTTP protocol implementation based on okhttp
URL: https://github.com/apache/nutch/pull/328
 
 
   A Nutch protocol plugin based on [okhttp](http://square.github.io/okhttp/):
   
   - derived from @jnioche's implementation for [storm-crawler#443](/DigitalPebble/storm-crawler/issues/443)
     - use okhttp's internal buffer for buffering content
   - adapted to be compatible to Nutch and behave almost the same as protocol-http
     - moved shared configuration settings to HttpBase (lib-http)
   - unit tests taken from protocol-http
   
   TODOs:
   - verify that issues reported in NUTCH-2549 do not appear again
   - complete unit tests
   - benchmark and large-scale test
   
   For HTTP/2 support a Java version at runtime is needed which supports ALPN:
   
   ```
   export NUTCH_JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64
   
   bin/nutch parsechecker \
      -Dplugin.includes='protocol-okhttp|parse-html' \
      -Dhttp.useHttp2=true \
      -Dstore.http.headers=true \
      https://www.google.com/
   ...
   _response.headers_=HTTP/2 200
   date: Wed, 09 May 2018 11:23:21 GMT
   expires: -1
   cache-control: private, max-age=0
   content-type: text/html; charset=ISO-8859-1
   p3p: CP="This is not a P3P policy! See g.co/p3phelp for more info."
   content-encoding: gzip
   server: gws
   x-xss-protection: 1; mode=block
   ...
   ```
   
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


> HTTP protocol plugin based on okhttp
> ------------------------------------
>
>                 Key: NUTCH-2576
>                 URL: https://issues.apache.org/jira/browse/NUTCH-2576
>             Project: Nutch
>          Issue Type: Improvement
>          Components: plugin, protocol
>            Reporter: Sebastian Nagel
>            Priority: Major
>             Fix For: 1.15
>
>
> [Okhttp|http://square.github.io/okhttp/] is an Apache2-licensed http library which supports HTTP/2. [~jnioche]'s implementation [storm-crawler#443|https://github.com/DigitalPebble/storm-crawler/issues/443] proves that it should be straightforward to implement a Nutch protocol plugin using okhttp. A recent HTTP protocol implementation should also fix (most of) the issues reported in NUTCH-2549.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)