You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@nutch.apache.org by sn...@apache.org on 2018/06/12 15:41:20 UTC
[nutch] branch master updated (b11082a -> 4bcaeeb)
This is an automated email from the ASF dual-hosted git repository.
snagel pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/nutch.git.
from b11082a Merge pull request #345 from sebastian-nagel/NUTCH-2595-crawler-commons-0.10
add 32860a5 NUTCH-2576 HTTP protocol implementation based on okhttp - derived from storm-crawler#443 (contributed by Julien Nioche)
add 659e1c8 NUTCH-2576 HTTP protocol implementation based on okhttp - fix: copy buffer(s) entirely to response (readFully vs. read)
add f598db7 NUTCH-2576 HTTP protocol implementation based on okhttp - do not catch IOException thrown by okhttp, to be caught by calling method (FetcherThread)
add dbdb40b NUTCH-2576 HTTP protocol implementation based on okhttp - set Cookie header (same as in protocol-http)
add 466a0ed NUTCH-2576 HTTP protocol implementation based on okhttp - port unit tests for NUTCH-2549 from protocol-http - ignore failing unit tests
add f1aa728 NUTCH-2576 HTTP protocol implementation based on okhttp - change port number of test server used for unit tests to avoid error with concurrently running tests of protocol-http
new 4bcaeeb Merge pull request #328 from sebastian-nagel/nutch-2576-protocol-okhttp
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
build.xml | 4 +
conf/nutch-default.xml | 22 ++
.../org/apache/nutch/metadata/HttpHeaders.java | 4 +
.../org/apache/nutch/net/protocols/Response.java | 26 ++
src/plugin/build.xml | 3 +
.../apache/nutch/protocol/http/api/HttpBase.java | 64 ++++-
.../nutch/protocol/htmlunit/HttpResponse.java | 4 +-
.../apache/nutch/protocol/http/HttpResponse.java | 23 +-
.../{protocol-http => protocol-okhttp}/build.xml | 2 +-
.../{index-metadata => protocol-okhttp}/ivy.xml | 1 +
.../jsp/basic-http.jsp | 0
.../jsp/brokenpage.jsp | 0
.../jsp/redirect301.jsp | 0
.../jsp/redirect302.jsp | 0
.../{protocol-http => protocol-okhttp}/plugin.xml | 22 +-
.../org/apache/nutch/protocol/okhttp/OkHttp.java | 248 ++++++++++++++++
.../nutch/protocol/okhttp/OkHttpResponse.java | 191 +++++++++++++
.../nutch/protocol/okhttp}/package-info.java | 5 +-
.../src/test/conf/nutch-site-test.xml | 21 +-
.../protocol/okhttp/TestBadServerResponses.java | 318 +++++++++++++++++++++
.../nutch/protocol/okhttp/TestProtocolOkHttp.java} | 9 +-
21 files changed, 912 insertions(+), 55 deletions(-)
copy src/plugin/{protocol-http => protocol-okhttp}/build.xml (97%)
copy src/plugin/{index-metadata => protocol-okhttp}/ivy.xml (94%)
copy src/plugin/{protocol-http => protocol-okhttp}/jsp/basic-http.jsp (100%)
copy src/plugin/{protocol-http => protocol-okhttp}/jsp/brokenpage.jsp (100%)
copy src/plugin/{protocol-http => protocol-okhttp}/jsp/redirect301.jsp (100%)
copy src/plugin/{protocol-http => protocol-okhttp}/jsp/redirect302.jsp (100%)
copy src/plugin/{protocol-http => protocol-okhttp}/plugin.xml (70%)
create mode 100755 src/plugin/protocol-okhttp/src/java/org/apache/nutch/protocol/okhttp/OkHttp.java
create mode 100644 src/plugin/protocol-okhttp/src/java/org/apache/nutch/protocol/okhttp/OkHttpResponse.java
copy src/{java/org/apache/nutch/scoring => plugin/protocol-okhttp/src/java/org/apache/nutch/protocol/okhttp}/package-info.java (83%)
copy src/plugin/{protocol-httpclient => protocol-okhttp}/src/test/conf/nutch-site-test.xml (75%)
create mode 100644 src/plugin/protocol-okhttp/src/test/org/apache/nutch/protocol/okhttp/TestBadServerResponses.java
copy src/plugin/{protocol-http/src/test/org/apache/nutch/protocol/http/TestProtocolHttp.java => protocol-okhttp/src/test/org/apache/nutch/protocol/okhttp/TestProtocolOkHttp.java} (96%)
--
To stop receiving notification emails like this one, please contact
snagel@apache.org.
[nutch] 01/01: Merge pull request #328 from
sebastian-nagel/nutch-2576-protocol-okhttp
Posted by sn...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
snagel pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nutch.git
commit 4bcaeeb8bfeaefadb627ea1a4eec56a9670ff777
Merge: b11082a f1aa728
Author: Sebastian Nagel <sn...@apache.org>
AuthorDate: Tue Jun 12 17:41:18 2018 +0200
Merge pull request #328 from sebastian-nagel/nutch-2576-protocol-okhttp
NUTCH-2576 HTTP protocol implementation based on okhttp
build.xml | 4 +
conf/nutch-default.xml | 22 ++
.../org/apache/nutch/metadata/HttpHeaders.java | 4 +
.../org/apache/nutch/net/protocols/Response.java | 26 ++
src/plugin/build.xml | 3 +
.../apache/nutch/protocol/http/api/HttpBase.java | 64 ++++-
.../nutch/protocol/htmlunit/HttpResponse.java | 4 +-
.../apache/nutch/protocol/http/HttpResponse.java | 23 +-
src/plugin/protocol-okhttp/build.xml | 50 ++++
src/plugin/protocol-okhttp/ivy.xml | 42 +++
src/plugin/protocol-okhttp/jsp/basic-http.jsp | 44 +++
src/plugin/protocol-okhttp/jsp/brokenpage.jsp | 47 +++
src/plugin/protocol-okhttp/jsp/redirect301.jsp | 49 ++++
src/plugin/protocol-okhttp/jsp/redirect302.jsp | 49 ++++
src/plugin/protocol-okhttp/plugin.xml | 53 ++++
.../org/apache/nutch/protocol/okhttp/OkHttp.java | 248 ++++++++++++++++
.../nutch/protocol/okhttp/OkHttpResponse.java | 191 +++++++++++++
.../nutch/protocol/okhttp/package-info.java} | 29 +-
.../src/test/conf/nutch-site-test.xml | 43 +++
.../protocol/okhttp/TestBadServerResponses.java | 318 +++++++++++++++++++++
.../nutch/protocol/okhttp/TestProtocolOkHttp.java | 141 +++++++++
21 files changed, 1406 insertions(+), 48 deletions(-)
--
To stop receiving notification emails like this one, please contact
snagel@apache.org.