You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2020/01/09 14:19:00 UTC
[jira] [Commented] (NUTCH-2733) protocol-okhttp: add support for
Brotli compression (Content-Encoding)
[ https://issues.apache.org/jira/browse/NUTCH-2733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17011864#comment-17011864 ]
ASF GitHub Bot commented on NUTCH-2733:
---------------------------------------
sebastian-nagel commented on pull request #492: NUTCH-2733 protocol-okhttp: add support for Brotli compression (Content-Encoding)
URL: https://github.com/apache/nutch/pull/492
- upgrade to okhttp 4.3.2
- add dependency to okhttp3.brotli and enable Brotli compression
```
$> bin/nutch parsechecker -Dstore.http.headers=true -Dstore.http.request=true \
-Dplugin.includes='protocol-okhttp|parse-html' https://www.youtube.com/
...
_request_=GET / HTTP/1.1
...
Accept-Encoding: br,gzip
...
_response.headers_=HTTP/1.1 200 OK
Content-Encoding: br
...
```
I'll run a test crawl with some additional metrics soon.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
> protocol-okhttp: add support for Brotli compression (Content-Encoding)
> ----------------------------------------------------------------------
>
> Key: NUTCH-2733
> URL: https://issues.apache.org/jira/browse/NUTCH-2733
> Project: Nutch
> Issue Type: Improvement
> Components: plugin, protocol
> Affects Versions: 1.15
> Reporter: Sebastian Nagel
> Priority: Major
> Labels: easytask, help-wanted, needs-test
> Fix For: 1.17
>
>
> [Okhttp 4.1.0|https://square.github.io/okhttp/changelog/#version-410] adds support for Brotli compression. We might upgrade the okhttp dependency to enable support for the "brotli" Content-Encoding.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)