You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by GitBox <gi...@apache.org> on 2022/06/15 12:47:37 UTC
[GitHub] [nutch] sebastian-nagel opened a new pull request, #733: NUTCH-2936 / NUTCH-2949 URLStreamHandler may fail jobs in distributed mode
sebastian-nagel opened a new pull request, #733:
URL: https://github.com/apache/nutch/pull/733
Fixes to address the failing of Nutch jobs in (pseudo-)distributed mode. Implements:
- caching of URLStreamHandlers per protocol to avoid that handlers are created anew
- enforce routing of standard protocols (http, https, file, jar) to handlers implemented by the JVM
- utilizes the URLStreamHandler cache
- fixes NUTCH-2936 (verified in pseudo-distributed mode)
Also:
- code improvements in classes of the package "org.apache.nutch.plugin"
- use `Class<?>` and remove suppressions of warnings
- javadocs: fix typos
- remove superfluous white space
- autoformat using code style template
- protocol-okhttp: initialize SSLContext not in a static code block (SSLContext is used to ignore SSL/TLS certificate verification): this was the initial fix for the needless warning in parsechecker even in local mode. This seems also fixed by the enforced routing of standard URLStreamHandlers, but I left it in, to avoid that all the testing in pseudo-distributed mode needs to be run again.
Next week I will test the fixes in real distributed mode.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@nutch.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nutch] lewismc commented on pull request #733: NUTCH-2936 / NUTCH-2949 URLStreamHandler may fail jobs in distributed mode
Posted by GitBox <gi...@apache.org>.
lewismc commented on PR #733:
URL: https://github.com/apache/nutch/pull/733#issuecomment-1157149313
This is exciting!!! Excellent debugging 👍 ... you got further than me.
I can't get around to testing it until next week at earliest.
Thinking back, I did observe revisits (recursive access) to URLStreamHandlerFactory but didn't pursue that line of inquiry at that point in time.
To get a bit more context I did review [HADOOP-14598-005.patch](https://issues.apache.org/jira/secure/attachment/12880380/HADOOP-14598-005.patch) and the current class it affects. Reading the code it makes more sense but admittedly until I debug this I still don't have the full context.
I took a look at [hadoop-hdfs TestUrlStreamHandler.java](https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestUrlStreamHandler.java) as well which I really like the look of. To build out some more confidence in this aspect of the codebase, we could create some tests for the [nutch URLStreamHandlerFactory.java](https://github.com/apache/nutch/blob/master/src/java/org/apache/nutch/plugin/URLStreamHandlerFactory.java).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@nutch.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nutch] sebastian-nagel commented on pull request #733: NUTCH-2936 / NUTCH-2949 URLStreamHandler may fail jobs in distributed mode
Posted by GitBox <gi...@apache.org>.
sebastian-nagel commented on PR #733:
URL: https://github.com/apache/nutch/pull/733#issuecomment-1165588041
Ok, this PR/patch was merged into [Common Crawl's "production" branch](/commoncrawl/nutch/) and successfully tested on a Hadoop cluster (3.2.3, Bigtop 3.1.0) of 30+ nodes and running various Nutch tools: inject, generate, fetch, updatedb, dedup, readdb.
@lewismc: I've opened NUTCH-2954 to address your suggestion to add unit tests for the URL stream handlers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@nutch.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nutch] sebastian-nagel merged pull request #733: NUTCH-2936 / NUTCH-2949 URLStreamHandler may fail jobs in distributed mode
Posted by GitBox <gi...@apache.org>.
sebastian-nagel merged PR #733:
URL: https://github.com/apache/nutch/pull/733
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@nutch.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org