You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by "igiguere (via GitHub)" <gi...@apache.org> on 2025/11/26 21:05:02 UTC
[PR] Fix for NUTCH-2671: Unt tests fail with JDK 17 [nutch]
igiguere opened a new pull request, #870:
URL: https://github.com/apache/nutch/pull/870
Fix for NUTCH-2671 contributed by igiguere. Also fixes NUTCH-3128, NUTCH-3125
This PR removes the dependency on MRunit.
Adding class ReducerContextWrapper to supply a minimal Context for testing, and using this class to replace the use of MRunit.
Removing class CrawlDbUpdateTestDriver, since only one test used it, and that test is now updated to use the new ReducerContextWrapper.
Unrelated improvements:
* adding an Ant target to get the dependency tree of test projects
* in TestProtocolHttpByProxy, look for an available port to start the proxy server
All unit tests pass locally:
* ant clean compile test
* ant test-full -Dtestcase=TestSegmentMerger
* ant test-full -Dtestcase=TestSegmentMergerCrawlDatums
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@nutch.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix for NUTCH-2971: Unt tests fail with JDK 17 [nutch]
Posted by "igiguere (via GitHub)" <gi...@apache.org>.
igiguere commented on PR #870:
URL: https://github.com/apache/nutch/pull/870#issuecomment-3589684424
@sebastian-nagel : I don't have permission to self-assign. I am logged in ASF Jira. I did subscribe to the Nutch dev mailing list. Anything missing?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@nutch.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix for NUTCH-2671: Unt tests fail with JDK 17 [nutch]
Posted by "igiguere (via GitHub)" <gi...@apache.org>.
igiguere commented on PR #870:
URL: https://github.com/apache/nutch/pull/870#issuecomment-3586749650
The 'run-rat' task is failing while downloading apache-rat-0.16.1-bin.tar.gz
I just ran the task locally without issue. Better luck next time, I guess.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@nutch.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix for NUTCH-2971: Unt tests fail with JDK 17 [nutch]
Posted by "igiguere (via GitHub)" <gi...@apache.org>.
igiguere closed pull request #870: Fix for NUTCH-2971: Unt tests fail with JDK 17
URL: https://github.com/apache/nutch/pull/870
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@nutch.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix for NUTCH-2971: Unt tests fail with JDK 17 [nutch]
Posted by "sebastian-nagel (via GitHub)" <gi...@apache.org>.
sebastian-nagel commented on PR #870:
URL: https://github.com/apache/nutch/pull/870#issuecomment-3589970711
> Anything missing?
Sorry, my fault. I have to do it. Thanks for trying!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@nutch.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix for NUTCH-2671: Unt tests fail with JDK 17 [nutch]
Posted by "sebastian-nagel (via GitHub)" <gi...@apache.org>.
sebastian-nagel commented on PR #870:
URL: https://github.com/apache/nutch/pull/870#issuecomment-3586977980
Thanks, @igiguere! This looks amazing, awesome work!
> Better luck next time, I guess.
Yes, indeed. Rerunning the job was successful.
> Fix for NUTCH-2671...
You mean NUTCH-2971, I guess.
> Also fixes NUTCH-3128, NUTCH-3125
This is great!
Give me one more day to have a closer look. But as far as I can see, this really a significant step forward. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@nutch.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix for NUTCH-2971: Unt tests fail with JDK 17 [nutch]
Posted by "sebastian-nagel (via GitHub)" <gi...@apache.org>.
sebastian-nagel commented on code in PR #870:
URL: https://github.com/apache/nutch/pull/870#discussion_r2571801576
##########
build.xml:
##########
@@ -86,6 +86,10 @@
<ivy:dependencytree />
</target>
+ <target name="dependencytests" depends="resolve-test" description="Show unit tests dependency tree">
Review Comment:
+1
##########
src/plugin/protocol-http/src/test/org/apache/nutch/protocol/http/TestProtocolHttpByProxy.java:
##########
@@ -30,26 +31,43 @@
import static org.junit.jupiter.api.Assertions.assertNotNull;
import static org.junit.jupiter.api.Assertions.assertTrue;
+import java.io.IOException;
+import java.net.ServerSocket;
+
/**
* Test cases for protocol-http by proxy
*/
public class TestProtocolHttpByProxy extends AbstractHttpProtocolPluginTest {
public static final String PROXY_HOST = "localhost";
- public static final Integer PROXY_PORT = 8888;
+ public Integer proxyPort = 8888;
public static final String TARGET_HOST = "www.baidu.com";
public static final Integer TARGET_PORT = 443;
+
+ private HttpProxyServer server;
@BeforeEach
public void setUp() throws Exception {
super.setUp();
+ proxyPort = findOpenPort();
conf.set("http.proxy.host", PROXY_HOST);
- conf.set("http.proxy.port", PROXY_PORT.toString());
+ conf.set("http.proxy.port", proxyPort.toString());
http.setConf(conf);
- HttpProxyServer server = DefaultHttpProxyServer.bootstrap()
- .withPort(PROXY_PORT).start();
+ server = DefaultHttpProxyServer.bootstrap()
+ .withPort(proxyPort).start();
+ }
+
+ private Integer findOpenPort() throws IOException {
Review Comment:
Great! This was an annoying issue, the protocol tests failing randomly from time to time because the port was occupied.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@nutch.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix for NUTCH-2971: Unt tests fail with JDK 17 [nutch]
Posted by "igiguere (via GitHub)" <gi...@apache.org>.
igiguere commented on PR #870:
URL: https://github.com/apache/nutch/pull/870#issuecomment-3587007355
> Thanks, @igiguere! This looks amazing, awesome work!
>
> > Better luck next time, I guess.
>
> Yes, indeed. Rerunning the job was successful.
>
> > Fix for NUTCH-2671...
>
> You mean [NUTCH-2971](https://issues.apache.org/jira/browse/NUTCH-2971), I guess.
>
> > Also fixes NUTCH-3128, NUTCH-3125
>
> This is great!
>
> Give me one more day to have a closer look. But as far as I can see, this really a significant step forward. Thanks!
Thanks. No worries. Just trying to keep busy
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@nutch.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix for NUTCH-2971: Unt tests fail with JDK 17 [nutch]
Posted by "sebastian-nagel (via GitHub)" <gi...@apache.org>.
sebastian-nagel merged PR #870:
URL: https://github.com/apache/nutch/pull/870
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@nutch.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org