You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2021/03/15 06:53:00 UTC
[GitHub] [hadoop] lichaojacobs opened a new pull request #2775: MAPREDUCE-7329: HadoopPipes task may fail when linux kernel version change from 3.x to 4.x
lichaojacobs opened a new pull request #2775:
URL: https://github.com/apache/hadoop/pull/2775
jira: https://issues.apache.org/jira/browse/MAPREDUCE-7329
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2775: MAPREDUCE-7329: HadoopPipes task has failed because of the ping timeout exception
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2775:
URL: https://github.com/apache/hadoop/pull/2775#issuecomment-815007568
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 36s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 2s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 22m 43s | | trunk passed |
| +1 :green_heart: | compile | 2m 36s | | trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 2m 11s | | trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 3s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 17s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 48s | | trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 42s | | trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 2m 12s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 5s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 25s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 3s | | the patch passed |
| +1 :green_heart: | compile | 2m 26s | | the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 2m 26s | | the patch passed |
| +1 :green_heart: | compile | 2m 4s | | the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | javac | 2m 4s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 54s | [/results-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2775/5/artifact/out/results-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt) | hadoop-mapreduce-project/hadoop-mapreduce-client: The patch generated 6 new + 7 unchanged - 0 fixed = 13 total (was 7) |
| +1 :green_heart: | mvnsite | 1m 6s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 34s | | the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 33s | | the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 2m 20s | | the patch passed |
| +1 :green_heart: | shadedclient | 16m 15s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 7m 50s | [/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2775/5/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt) | hadoop-mapreduce-client-core in the patch passed. |
| +1 :green_heart: | unit | 134m 28s | | hadoop-mapreduce-client-jobclient in the patch passed. |
| -1 :x: | asflicense | 0m 38s | [/results-asflicense.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2775/5/artifact/out/results-asflicense.txt) | The patch generated 1 ASF License warnings. |
| | | 236m 48s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.mapred.TestJobEndNotifier |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2775/5/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2775 |
| JIRA Issue | MAPREDUCE-7329 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 604906062d2e 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 459bdcdfc42dc481e5796fbe0d9896a244382c45 |
| Default Java | Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2775/5/testReport/ |
| Max. process+thread count | 1559 (vs. ulimit of 5500) |
| modules | C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2775/5/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] lichaojacobs edited a comment on pull request #2775: MAPREDUCE-7329: HadoopPipes task may fail when linux kernel upgrade from 3.x to 4.x
Posted by GitBox <gi...@apache.org>.
lichaojacobs edited a comment on pull request #2775:
URL: https://github.com/apache/hadoop/pull/2775#issuecomment-799868579
@liuml07 @steveloughran Could you please help review the patch? It aims to fix Hadoop Pipes bug. Thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] lichaojacobs commented on pull request #2775: MAPREDUCE-7329: HadoopPipes task may fail when linux kernel upgrade from 3.x to 4.x
Posted by GitBox <gi...@apache.org>.
lichaojacobs commented on pull request #2775:
URL: https://github.com/apache/hadoop/pull/2775#issuecomment-800473749
@aajisaka Could you please help review the patch? thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] lichaojacobs commented on pull request #2775: MAPREDUCE-7329: HadoopPipes task has failed because of the ping timeout exception
Posted by GitBox <gi...@apache.org>.
lichaojacobs commented on pull request #2775:
URL: https://github.com/apache/hadoop/pull/2775#issuecomment-814661823
> Can `ServerSocket#setSoTimeout(int)` be used instead of modifying the kernel parameter?
Of course, but need to accept client socket first? In my cleaner implement, it will close socket correctly, and thus accept queue will also be cleared.
And there is another problem, if we set so timeout, then which value is suitable? If too short, data stream may block due to gc or other reasons which will cause task failed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] lichaojacobs commented on a change in pull request #2775: MAPREDUCE-7329: HadoopPipes task has failed because of the ping timeout exception
Posted by GitBox <gi...@apache.org>.
lichaojacobs commented on a change in pull request #2775:
URL: https://github.com/apache/hadoop/pull/2775#discussion_r603756448
##########
File path: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/pipes/Application.java
##########
@@ -266,4 +271,39 @@ public static String createDigest(byte[] password, String data)
return SecureShuffleUtils.hashFromString(data, key);
}
+ private class PingSocketCleaner extends Thread {
+ PingSocketCleaner(String name) {
+ super(name);
+ }
+
+ @Override
+ public void run() {
+ LOG.info("PingSocketCleaner started...");
+ while (true) {
+ Socket clientSocket = null;
+ try {
+ clientSocket = serverSocket.accept();
+ if (LOG.isDebugEnabled()) {
Review comment:
ok , i am just used to using `isDebugEnabled `, and in this case, it's really not nesscessary
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2775: MAPREDUCE-7329: HadoopPipes task may fail when linux kernel version change from 3.x to 4.x
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2775:
URL: https://github.com/apache/hadoop/pull/2775#issuecomment-799212430
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 36s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 34m 9s | | trunk passed |
| +1 :green_heart: | compile | 0m 44s | | trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 0m 35s | | trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 34s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 39s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 22s | | trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 20s | | trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 19s | | trunk passed |
| +1 :green_heart: | shadedclient | 14m 50s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 32s | | the patch passed |
| +1 :green_heart: | compile | 0m 34s | | the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 0m 34s | | the patch passed |
| +1 :green_heart: | compile | 0m 28s | | the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | javac | 0m 28s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 27s | [/results-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2775/1/artifact/out/results-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt) | hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core: The patch generated 1 new + 7 unchanged - 0 fixed = 8 total (was 7) |
| +1 :green_heart: | mvnsite | 0m 32s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 17s | | the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 14s | | the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 20s | | the patch passed |
| +1 :green_heart: | shadedclient | 14m 28s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 7m 12s | | hadoop-mapreduce-client-core in the patch passed. |
| +1 :green_heart: | asflicense | 0m 33s | | The patch does not generate ASF License warnings. |
| | | 81m 16s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2775/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2775 |
| JIRA Issue | MAPREDUCE-7329 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux bc05a6db4771 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 393b71c6182645f660d87b74eb6e2d745200f5b5 |
| Default Java | Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2775/1/testReport/ |
| Max. process+thread count | 1612 (vs. ulimit of 5500) |
| modules | C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2775/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] aajisaka commented on a change in pull request #2775: MAPREDUCE-7329: HadoopPipes task has failed because of the ping timeout exception
Posted by GitBox <gi...@apache.org>.
aajisaka commented on a change in pull request #2775:
URL: https://github.com/apache/hadoop/pull/2775#discussion_r608365538
##########
File path: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/pipes/Application.java
##########
@@ -133,6 +135,10 @@
process = runClient(cmd, env);
clientSocket = serverSocket.accept();
+ // start ping socket cleaner
+ socketCleaner = new PingSocketCleaner("ping-socket-cleaner");
+ socketCleaner.setDaemon(true);
+ socketCleaner.start();
Review comment:
The cleaner thread is never stopped. Would you clean up the thread in the cleanup method?
##########
File path: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/pipes/Application.java
##########
@@ -266,4 +272,32 @@ public static String createDigest(byte[] password, String data)
return SecureShuffleUtils.hashFromString(data, key);
}
+ private class PingSocketCleaner extends Thread {
+ PingSocketCleaner(String name) {
+ super(name);
+ }
+
+ @Override
+ public void run() {
+ LOG.info("PingSocketCleaner started...");
+ while (true) {
+ Socket clientSocket = null;
+ try {
+ clientSocket = serverSocket.accept();
+ LOG.debug("Connection received from {}", clientSocket.getInetAddress());
+ int readData = 0;
+ while (readData != -1) {
+ readData = clientSocket.getInputStream().read();
+ }
+ LOG.debug("close socket cause client has closed.");
+ clientSocket.close();
+ } catch (IOException exception) {
+ LOG.error("PingSocketCleaner exception", exception);
+ IOUtils.closeSocket(clientSocket);
Review comment:
Do not need this line. `IOUtils.closeSocket` in the finally clause is always executed.
```suggestion
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] aajisaka commented on a change in pull request #2775: MAPREDUCE-7329: HadoopPipes task has failed because of the ping timeout exception
Posted by GitBox <gi...@apache.org>.
aajisaka commented on a change in pull request #2775:
URL: https://github.com/apache/hadoop/pull/2775#discussion_r609446125
##########
File path: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/pipes/TestPipeApplication.java
##########
@@ -455,6 +460,96 @@ public void testPipesPartitioner() {
assertEquals(3, partitioner.getPartition(iw, new Text("test"), 2));
}
+ @Test
+ public void testSocketCleaner() throws Exception {
+ ServerSocket serverSocket = setupServerSocket();
+ SocketCleaner cleaner = setupCleaner(serverSocket);
+ // mock ping thread, connect to server socket per second.
+ int expectedClosedCount = 5;
+ for (int i = 0; i < expectedClosedCount; i++) {
+ try {
+ Thread.sleep(1000);
+ Socket clientSocket = new Socket(serverSocket.getInetAddress(),
+ serverSocket.getLocalPort());
+ clientSocket.close();
+ } catch (Exception exception) {
+ // ignored...
+ exception.printStackTrace();
+ }
+ }
+ waitUntilExhausted(
+ () -> expectedClosedCount == cleaner.getCloseSocketCount(), 5000);
+ }
+
+ @Test
+ public void testSocketTimeout() throws Exception {
+ ServerSocket serverSocket = setupServerSocket();
+ SocketCleaner cleaner = setupCleaner(serverSocket, 100);
+ try {
+ new Socket(serverSocket.getInetAddress(), serverSocket.getLocalPort());
+ Thread.sleep(1000);
+ } catch (Exception exception) {
+ // ignored...
+ }
+ waitUntilExhausted(() -> 1 == cleaner.getCloseSocketCount(), 5000);
Review comment:
`GenericTestUtils.waitFor` can be used.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2775: MAPREDUCE-7329: HadoopPipes task has failed because of the ping timeout exception
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2775:
URL: https://github.com/apache/hadoop/pull/2775#issuecomment-815020952
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 49s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 13m 48s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 26m 0s | | trunk passed |
| +1 :green_heart: | compile | 2m 59s | | trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 2m 28s | | trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 1s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 18s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 46s | | trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 42s | | trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 2m 19s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 46s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 20s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 2s | | the patch passed |
| +1 :green_heart: | compile | 2m 33s | | the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 2m 33s | | the patch passed |
| +1 :green_heart: | compile | 2m 4s | | the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | javac | 2m 4s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 52s | [/results-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2775/6/artifact/out/results-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt) | hadoop-mapreduce-project/hadoop-mapreduce-client: The patch generated 11 new + 7 unchanged - 0 fixed = 18 total (was 7) |
| +1 :green_heart: | mvnsite | 1m 1s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 32s | | the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 30s | | the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 2m 13s | | the patch passed |
| +1 :green_heart: | shadedclient | 16m 10s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 7m 39s | [/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2775/6/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt) | hadoop-mapreduce-client-core in the patch passed. |
| +1 :green_heart: | unit | 130m 25s | | hadoop-mapreduce-client-jobclient in the patch passed. |
| -1 :x: | asflicense | 0m 35s | [/results-asflicense.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2775/6/artifact/out/results-asflicense.txt) | The patch generated 1 ASF License warnings. |
| | | 236m 43s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.mapred.TestJobEndNotifier |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2775/6/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2775 |
| JIRA Issue | MAPREDUCE-7329 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux cb7456b0eb4c 4.15.0-126-generic #129-Ubuntu SMP Mon Nov 23 18:53:38 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 2012f3038f5ef44bb1db62cd239927503b1c4bf1 |
| Default Java | Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2775/6/testReport/ |
| Max. process+thread count | 1176 (vs. ulimit of 5500) |
| modules | C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2775/6/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2775: MAPREDUCE-7329: HadoopPipes task has failed because of the ping timeout exception
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2775:
URL: https://github.com/apache/hadoop/pull/2775#issuecomment-809906960
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 36s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 40s | | trunk passed |
| +1 :green_heart: | compile | 0m 41s | | trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 0m 36s | | trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 35s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 44s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 24s | | trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 21s | | trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 20s | | trunk passed |
| +1 :green_heart: | shadedclient | 14m 32s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 35s | | the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 0m 35s | | the patch passed |
| +1 :green_heart: | compile | 0m 27s | | the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | javac | 0m 27s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 26s | [/results-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2775/2/artifact/out/results-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt) | hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core: The patch generated 2 new + 7 unchanged - 0 fixed = 9 total (was 7) |
| +1 :green_heart: | mvnsite | 0m 31s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 15s | | the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 14s | | the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 19s | | the patch passed |
| +1 :green_heart: | shadedclient | 14m 23s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 7m 15s | [/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2775/2/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt) | hadoop-mapreduce-client-core in the patch passed. |
| +1 :green_heart: | asflicense | 0m 32s | | The patch does not generate ASF License warnings. |
| | | 80m 27s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.mapred.TestJobEndNotifier |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2775/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2775 |
| JIRA Issue | MAPREDUCE-7329 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 2bf8a2b04881 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 5dcb6055edccc1f12c2531002b7b66a611d9761f |
| Default Java | Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2775/2/testReport/ |
| Max. process+thread count | 1557 (vs. ulimit of 5500) |
| modules | C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2775/2/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] lichaojacobs commented on a change in pull request #2775: MAPREDUCE-7329: HadoopPipes task has failed because of the ping timeout exception
Posted by GitBox <gi...@apache.org>.
lichaojacobs commented on a change in pull request #2775:
URL: https://github.com/apache/hadoop/pull/2775#discussion_r608382114
##########
File path: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/pipes/Application.java
##########
@@ -133,6 +135,10 @@
process = runClient(cmd, env);
clientSocket = serverSocket.accept();
+ // start ping socket cleaner
+ socketCleaner = new PingSocketCleaner("ping-socket-cleaner");
+ socketCleaner.setDaemon(true);
+ socketCleaner.start();
Review comment:
ok, i will do that in cleanup method
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2775: MAPREDUCE-7329: HadoopPipes task has failed because of the ping timeout exception
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2775:
URL: https://github.com/apache/hadoop/pull/2775#issuecomment-811633948
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 35s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 12s | | trunk passed |
| +1 :green_heart: | compile | 0m 39s | | trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 0m 36s | | trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 35s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 42s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 26s | | trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 24s | | trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 15s | | trunk passed |
| +1 :green_heart: | shadedclient | 14m 19s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 31s | | the patch passed |
| +1 :green_heart: | compile | 0m 31s | | the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 0m 31s | | the patch passed |
| +1 :green_heart: | compile | 0m 28s | | the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | javac | 0m 28s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 26s | [/results-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2775/4/artifact/out/results-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt) | hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core: The patch generated 2 new + 7 unchanged - 0 fixed = 9 total (was 7) |
| +1 :green_heart: | mvnsite | 0m 31s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 16s | | the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 16s | | the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 17s | | the patch passed |
| +1 :green_heart: | shadedclient | 14m 5s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 7m 29s | [/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2775/4/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt) | hadoop-mapreduce-client-core in the patch passed. |
| +1 :green_heart: | asflicense | 0m 33s | | The patch does not generate ASF License warnings. |
| | | 79m 48s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.mapred.TestJobEndNotifier |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2775/4/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2775 |
| JIRA Issue | MAPREDUCE-7329 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux b26c9dc0913a 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 9fce5d15c8457e30c02c31422458ee7d8debbe13 |
| Default Java | Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2775/4/testReport/ |
| Max. process+thread count | 1588 (vs. ulimit of 5500) |
| modules | C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2775/4/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] lichaojacobs commented on pull request #2775: MAPREDUCE-7329: HadoopPipes task has failed because of the ping timeout exception
Posted by GitBox <gi...@apache.org>.
lichaojacobs commented on pull request #2775:
URL: https://github.com/apache/hadoop/pull/2775#issuecomment-814649507
> Can we set a shorter timeout than the default (2h) in the server socket to clear the queue?
if do this, we need to modify kernel parameter `net.ipv4.tcp_keepalive_time` which i think is out of application scope and also dangerous
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] aajisaka merged pull request #2775: MAPREDUCE-7329: HadoopPipes task has failed because of the ping timeout exception
Posted by GitBox <gi...@apache.org>.
aajisaka merged pull request #2775:
URL: https://github.com/apache/hadoop/pull/2775
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] lichaojacobs commented on pull request #2775: MAPREDUCE-7329: HadoopPipes task has failed because of the ping timeout exception
Posted by GitBox <gi...@apache.org>.
lichaojacobs commented on pull request #2775:
URL: https://github.com/apache/hadoop/pull/2775#issuecomment-814788086
> > And we should do like this:
>
> LGTM
>
> I thought only setting so_timeout fixes this problem. However, it is not true. The problem is that the Application does not call `serverSocket.accept()` for ping requests to drain the tcp accept queue. Agreed to implement the ping socket cleaner. Thanks.
Yes, this is exactly what i am trying to fix. I will try to update with a regression test soon. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2775: MAPREDUCE-7329: HadoopPipes task has failed because of the ping timeout exception
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2775:
URL: https://github.com/apache/hadoop/pull/2775#issuecomment-815727225
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 34s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 15s | | Maven dependency ordering for branch |
| -1 :x: | mvninstall | 6m 4s | [/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2775/11/artifact/out/branch-mvninstall-root.txt) | root in trunk failed. |
| +1 :green_heart: | compile | 2m 24s | | trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 2m 0s | | trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 55s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 15s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 50s | | trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 42s | | trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 54s | | trunk passed |
| +1 :green_heart: | shadedclient | 13m 49s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 25s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 1s | | the patch passed |
| +1 :green_heart: | compile | 2m 17s | | the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 2m 17s | | the patch passed |
| +1 :green_heart: | compile | 1m 57s | | the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | javac | 1m 57s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 49s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 1s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 37s | | the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 34s | | the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 2m 7s | | the patch passed |
| +1 :green_heart: | shadedclient | 13m 38s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 7m 27s | [/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2775/11/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt) | hadoop-mapreduce-client-core in the patch passed. |
| +1 :green_heart: | unit | 131m 12s | | hadoop-mapreduce-client-jobclient in the patch passed. |
| +1 :green_heart: | asflicense | 0m 39s | | The patch does not generate ASF License warnings. |
| | | 210m 9s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.mapred.TestJobEndNotifier |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2775/11/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2775 |
| JIRA Issue | MAPREDUCE-7329 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 9228899b8f35 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 08790b1668caaf825fd79a7aff29c5e5e169ec45 |
| Default Java | Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2775/11/testReport/ |
| Max. process+thread count | 1639 (vs. ulimit of 5500) |
| modules | C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2775/11/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] aajisaka commented on pull request #2775: MAPREDUCE-7329: HadoopPipes task has failed because of the ping timeout exception
Posted by GitBox <gi...@apache.org>.
aajisaka commented on pull request #2775:
URL: https://github.com/apache/hadoop/pull/2775#issuecomment-814645217
Can we set a shorter timeout than the default (2h) in the server socket to clear the queue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] aajisaka commented on pull request #2775: MAPREDUCE-7329: HadoopPipes task has failed because of the ping timeout exception
Posted by GitBox <gi...@apache.org>.
aajisaka commented on pull request #2775:
URL: https://github.com/apache/hadoop/pull/2775#issuecomment-814776572
> And we should do like this:
LGTM
I thought only setting so_timeout fixes this problem. However, it is not true. The problem is that the Application does not call `serverSocket.accept()` for ping requests to drain the tcp accept queue. Agreed to implement the ping socket cleaner. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2775: MAPREDUCE-7329: HadoopPipes task has failed because of the ping timeout exception
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2775:
URL: https://github.com/apache/hadoop/pull/2775#issuecomment-815232495
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 50s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 5s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 22m 31s | | trunk passed |
| +1 :green_heart: | compile | 2m 49s | | trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 2m 5s | | trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 55s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 10s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 42s | | trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 39s | | trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 57s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 8s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 20s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 0m 59s | | the patch passed |
| +1 :green_heart: | compile | 2m 25s | | the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 2m 25s | | the patch passed |
| +1 :green_heart: | compile | 2m 3s | | the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | javac | 2m 3s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 51s | [/results-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2775/8/artifact/out/results-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt) | hadoop-mapreduce-project/hadoop-mapreduce-client: The patch generated 9 new + 39 unchanged - 0 fixed = 48 total (was 39) |
| +1 :green_heart: | mvnsite | 0m 58s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 31s | | the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 29s | | the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 2m 9s | | the patch passed |
| +1 :green_heart: | shadedclient | 16m 11s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 7m 40s | [/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2775/8/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt) | hadoop-mapreduce-client-core in the patch passed. |
| +1 :green_heart: | unit | 130m 30s | | hadoop-mapreduce-client-jobclient in the patch passed. |
| +1 :green_heart: | asflicense | 0m 35s | | The patch does not generate ASF License warnings. |
| | | 231m 14s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.mapred.TestJobEndNotifier |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2775/8/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2775 |
| JIRA Issue | MAPREDUCE-7329 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux c2e92af0fe23 4.15.0-126-generic #129-Ubuntu SMP Mon Nov 23 18:53:38 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / f43e450fede35d6eef0f36c6aaa9dc64a7a80d6b |
| Default Java | Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2775/8/testReport/ |
| Max. process+thread count | 1197 (vs. ulimit of 5500) |
| modules | C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2775/8/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] aajisaka commented on pull request #2775: MAPREDUCE-7329: HadoopPipes task has failed because of the ping timeout exception
Posted by GitBox <gi...@apache.org>.
aajisaka commented on pull request #2775:
URL: https://github.com/apache/hadoop/pull/2775#issuecomment-816366019
Merged. Thank you @lichaojacobs and @steveloughran
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] liuml07 commented on pull request #2775: MAPREDUCE-7329: HadoopPipes task may fail when linux kernel upgrade from 3.x to 4.x
Posted by GitBox <gi...@apache.org>.
liuml07 commented on pull request #2775:
URL: https://github.com/apache/hadoop/pull/2775#issuecomment-799906569
I did not touch this pipe class or package. I can take a look later this week, but @aajisaka and @wangdatan will have more context. Thanks,
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] lichaojacobs edited a comment on pull request #2775: MAPREDUCE-7329: HadoopPipes task has failed because of the ping timeout exception
Posted by GitBox <gi...@apache.org>.
lichaojacobs edited a comment on pull request #2775:
URL: https://github.com/apache/hadoop/pull/2775#issuecomment-815739108
It seems like all comments are addressed, and the `-1 ❌ | mvninstall` is also not related with my patch. FYI @aajisaka
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2775: MAPREDUCE-7329: HadoopPipes task has failed because of the ping timeout exception
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2775:
URL: https://github.com/apache/hadoop/pull/2775#issuecomment-815225279
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 37s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 36s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 21m 4s | | trunk passed |
| +1 :green_heart: | compile | 2m 30s | | trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 2m 8s | | trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 59s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 15s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 49s | | trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 42s | | trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 2m 4s | | trunk passed |
| +1 :green_heart: | shadedclient | 14m 7s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 30s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 21s | | the patch passed |
| +1 :green_heart: | compile | 2m 28s | | the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 2m 28s | | the patch passed |
| +1 :green_heart: | compile | 2m 0s | | the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | javac | 2m 0s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 50s | [/results-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2775/7/artifact/out/results-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt) | hadoop-mapreduce-project/hadoop-mapreduce-client: The patch generated 9 new + 39 unchanged - 0 fixed = 48 total (was 39) |
| +1 :green_heart: | mvnsite | 1m 3s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 36s | | the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 30s | | the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 2m 12s | | the patch passed |
| +1 :green_heart: | shadedclient | 13m 47s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 7m 43s | [/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2775/7/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt) | hadoop-mapreduce-client-core in the patch passed. |
| +1 :green_heart: | unit | 130m 50s | | hadoop-mapreduce-client-jobclient in the patch passed. |
| +1 :green_heart: | asflicense | 0m 38s | | The patch does not generate ASF License warnings. |
| | | 227m 21s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.mapred.TestJobEndNotifier |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2775/7/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2775 |
| JIRA Issue | MAPREDUCE-7329 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 757c8e1b496b 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / f43e450fede35d6eef0f36c6aaa9dc64a7a80d6b |
| Default Java | Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2775/7/testReport/ |
| Max. process+thread count | 1581 (vs. ulimit of 5500) |
| modules | C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2775/7/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] lichaojacobs commented on a change in pull request #2775: MAPREDUCE-7329: HadoopPipes task has failed because of the ping timeout exception
Posted by GitBox <gi...@apache.org>.
lichaojacobs commented on a change in pull request #2775:
URL: https://github.com/apache/hadoop/pull/2775#discussion_r609454979
##########
File path: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/pipes/TestPipeApplication.java
##########
@@ -455,6 +460,96 @@ public void testPipesPartitioner() {
assertEquals(3, partitioner.getPartition(iw, new Text("test"), 2));
}
+ @Test
+ public void testSocketCleaner() throws Exception {
+ ServerSocket serverSocket = setupServerSocket();
+ SocketCleaner cleaner = setupCleaner(serverSocket);
+ // mock ping thread, connect to server socket per second.
+ int expectedClosedCount = 5;
+ for (int i = 0; i < expectedClosedCount; i++) {
+ try {
+ Thread.sleep(1000);
+ Socket clientSocket = new Socket(serverSocket.getInetAddress(),
+ serverSocket.getLocalPort());
+ clientSocket.close();
+ } catch (Exception exception) {
+ // ignored...
+ exception.printStackTrace();
+ }
+ }
+ waitUntilExhausted(
+ () -> expectedClosedCount == cleaner.getCloseSocketCount(), 5000);
Review comment:
Thanks for advice!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #2775: MAPREDUCE-7329: HadoopPipes task may fail when linux kernel upgrade from 3.x to 4.x
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #2775:
URL: https://github.com/apache/hadoop/pull/2775#issuecomment-801006630
Never seen this code before so I'm not really in a position to review. Just trying to revise my sockets API knowledge, which dates from when I was writing Windows 3.1 code and hasn't been refreshed much, not since HTTP came along
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] lichaojacobs commented on a change in pull request #2775: MAPREDUCE-7329: HadoopPipes task has failed because of the ping timeout exception
Posted by GitBox <gi...@apache.org>.
lichaojacobs commented on a change in pull request #2775:
URL: https://github.com/apache/hadoop/pull/2775#discussion_r603752872
##########
File path: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/pipes/Application.java
##########
@@ -266,4 +271,39 @@ public static String createDigest(byte[] password, String data)
return SecureShuffleUtils.hashFromString(data, key);
}
+ private class PingSocketCleaner extends Thread {
+ PingSocketCleaner(String name) {
+ super(name);
+ }
+
+ @Override
+ public void run() {
+ LOG.info("PingSocketCleaner started...");
+ while (true) {
+ Socket clientSocket = null;
+ try {
+ clientSocket = serverSocket.accept();
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("Got one client socket...");
+ }
+ int readData = clientSocket.getInputStream().read();
Review comment:
this is just for ping socket, since ping client won't send data but just make sure server can connect. we can do this because we know client behavior. And yes, if other processes just connect server and send byte on purpose, the cleaner won't work well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #2775: MAPREDUCE-7329: HadoopPipes task has failed because of the ping timeout exception
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #2775:
URL: https://github.com/apache/hadoop/pull/2775#discussion_r595924047
##########
File path: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/pipes/Application.java
##########
@@ -266,4 +271,39 @@ public static String createDigest(byte[] password, String data)
return SecureShuffleUtils.hashFromString(data, key);
}
+ private class PingSocketCleaner extends Thread {
+ PingSocketCleaner(String name) {
+ super(name);
+ }
+
+ @Override
+ public void run() {
+ LOG.info("PingSocketCleaner started...");
+ while (true) {
+ Socket clientSocket = null;
+ try {
+ clientSocket = serverSocket.accept();
+ if (LOG.isDebugEnabled()) {
Review comment:
SLF4J doesn't need the isDebugEnabled() wrappers unless the log is doing any complex work. But here it might be good to keep and have the log also note the socket address, e,g
```java
LOG.debug("Connection received from {}", clientSocket.getInetAddress());
```
##########
File path: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/pipes/Application.java
##########
@@ -266,4 +271,39 @@ public static String createDigest(byte[] password, String data)
return SecureShuffleUtils.hashFromString(data, key);
}
+ private class PingSocketCleaner extends Thread {
+ PingSocketCleaner(String name) {
+ super(name);
+ }
+
+ @Override
+ public void run() {
+ LOG.info("PingSocketCleaner started...");
+ while (true) {
+ Socket clientSocket = null;
+ try {
+ clientSocket = serverSocket.accept();
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("Got one client socket...");
+ }
+ int readData = clientSocket.getInputStream().read();
Review comment:
so we accept() a connection, then read() one byte. If there is no data or there's an IOE, the socket is closed. But what if there is a byte()? what happens to the clientSocket?
##########
File path: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/pipes/Application.java
##########
@@ -266,4 +271,39 @@ public static String createDigest(byte[] password, String data)
return SecureShuffleUtils.hashFromString(data, key);
}
+ private class PingSocketCleaner extends Thread {
+ PingSocketCleaner(String name) {
+ super(name);
+ }
+
+ @Override
+ public void run() {
+ LOG.info("PingSocketCleaner started...");
+ while (true) {
+ Socket clientSocket = null;
+ try {
+ clientSocket = serverSocket.accept();
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("Got one client socket...");
+ }
+ int readData = clientSocket.getInputStream().read();
+ if (readData == -1) {
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("close socket cause client has closed.");
+ }
+ clientSocket.close();
+ }
+ } catch (IOException exception) {
+ LOG.error("PingSocketCleaner exception", exception);
+ if (clientSocket != null) {
Review comment:
use org.apache.hadoop.io.IOUtils.cleanupWithLogger()
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] aajisaka commented on a change in pull request #2775: MAPREDUCE-7329: HadoopPipes task has failed because of the ping timeout exception
Posted by GitBox <gi...@apache.org>.
aajisaka commented on a change in pull request #2775:
URL: https://github.com/apache/hadoop/pull/2775#discussion_r609445400
##########
File path: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/pipes/TestPipeApplication.java
##########
@@ -455,6 +460,96 @@ public void testPipesPartitioner() {
assertEquals(3, partitioner.getPartition(iw, new Text("test"), 2));
}
+ @Test
+ public void testSocketCleaner() throws Exception {
+ ServerSocket serverSocket = setupServerSocket();
+ SocketCleaner cleaner = setupCleaner(serverSocket);
+ // mock ping thread, connect to server socket per second.
+ int expectedClosedCount = 5;
+ for (int i = 0; i < expectedClosedCount; i++) {
+ try {
+ Thread.sleep(1000);
+ Socket clientSocket = new Socket(serverSocket.getInetAddress(),
+ serverSocket.getLocalPort());
+ clientSocket.close();
+ } catch (Exception exception) {
+ // ignored...
+ exception.printStackTrace();
+ }
+ }
+ waitUntilExhausted(
+ () -> expectedClosedCount == cleaner.getCloseSocketCount(), 5000);
Review comment:
GenericTestUtils can be used instead.
```suggestion
GenericTestUtils.waitFor(
() -> expectedClosedCount == cleaner.getCloseSocketCount(), 100, 5000);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] aajisaka commented on pull request #2775: MAPREDUCE-7329: HadoopPipes task has failed because of the ping timeout exception
Posted by GitBox <gi...@apache.org>.
aajisaka commented on pull request #2775:
URL: https://github.com/apache/hadoop/pull/2775#issuecomment-814684423
> Of course, but need to accept client socket first?
No, this method must be called before accepting client sockets.
> if we set so timeout, then which value is suitable?
The default value is 60 seconds in Hadoop IPC client (ipc.ping.interval). Is it too long for Hadoop Pipes application?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] aajisaka commented on pull request #2775: MAPREDUCE-7329: HadoopPipes task has failed because of the ping timeout exception
Posted by GitBox <gi...@apache.org>.
aajisaka commented on pull request #2775:
URL: https://github.com/apache/hadoop/pull/2775#issuecomment-814629478
Can we create a regression test for this issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] lichaojacobs commented on pull request #2775: MAPREDUCE-7329: HadoopPipes task has failed because of the ping timeout exception
Posted by GitBox <gi...@apache.org>.
lichaojacobs commented on pull request #2775:
URL: https://github.com/apache/hadoop/pull/2775#issuecomment-815739108
It seems like all comments are addressed, and the `-1 ❌ | mvninstall` is also not related with my patch. FYI @aajisaka , thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2775: MAPREDUCE-7329: HadoopPipes task has failed because of the ping timeout exception
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2775:
URL: https://github.com/apache/hadoop/pull/2775#issuecomment-809912756
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 36s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 9s | | trunk passed |
| +1 :green_heart: | compile | 0m 38s | | trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 0m 34s | | trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 34s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 40s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 23s | | trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 22s | | trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 20s | | trunk passed |
| +1 :green_heart: | shadedclient | 14m 18s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 32s | | the patch passed |
| +1 :green_heart: | compile | 0m 32s | | the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 0m 32s | | the patch passed |
| +1 :green_heart: | compile | 0m 28s | | the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | javac | 0m 28s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 25s | [/results-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2775/3/artifact/out/results-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt) | hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core: The patch generated 2 new + 7 unchanged - 0 fixed = 9 total (was 7) |
| +1 :green_heart: | mvnsite | 0m 33s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 15s | | the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 15s | | the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 24s | | the patch passed |
| +1 :green_heart: | shadedclient | 14m 21s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 7m 30s | [/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2775/3/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt) | hadoop-mapreduce-client-core in the patch passed. |
| +1 :green_heart: | asflicense | 0m 33s | | The patch does not generate ASF License warnings. |
| | | 79m 48s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.mapred.TestJobEndNotifier |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2775/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2775 |
| JIRA Issue | MAPREDUCE-7329 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux ac69e3c3a4b2 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / c4afa6873a5d9ff8927228ff5d1bb5c0b7692581 |
| Default Java | Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2775/3/testReport/ |
| Max. process+thread count | 1581 (vs. ulimit of 5500) |
| modules | C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2775/3/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #2775: MAPREDUCE-7329: HadoopPipes task has failed because of the ping timeout exception
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #2775:
URL: https://github.com/apache/hadoop/pull/2775#discussion_r605091729
##########
File path: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/pipes/Application.java
##########
@@ -266,4 +271,39 @@ public static String createDigest(byte[] password, String data)
return SecureShuffleUtils.hashFromString(data, key);
}
+ private class PingSocketCleaner extends Thread {
+ PingSocketCleaner(String name) {
+ super(name);
+ }
+
+ @Override
+ public void run() {
+ LOG.info("PingSocketCleaner started...");
+ while (true) {
+ Socket clientSocket = null;
+ try {
+ clientSocket = serverSocket.accept();
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("Got one client socket...");
+ }
+ int readData = clientSocket.getInputStream().read();
Review comment:
...and presumably a finally() round the read and use IOUtils.closeQuietly() for clientsocket so the case where it won't close because of some inner problem (such as it being closed:) don't trigger failures....
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] lichaojacobs commented on pull request #2775: MAPREDUCE-7329: HadoopPipes task may fail when linux kernel upgrade from 3.x to 4.x
Posted by GitBox <gi...@apache.org>.
lichaojacobs commented on pull request #2775:
URL: https://github.com/apache/hadoop/pull/2775#issuecomment-799868579
@liuml07 Could you please help review the patch? It aims to fix Hadoop Pipes bug. Thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] lichaojacobs commented on pull request #2775: MAPREDUCE-7329: HadoopPipes task may fail when linux kernel upgrade from 3.x to 4.x
Posted by GitBox <gi...@apache.org>.
lichaojacobs commented on pull request #2775:
URL: https://github.com/apache/hadoop/pull/2775#issuecomment-799987235
> I did not touch this pipe class or package. I can take a look later this week, but @aajisaka and @wangdatan will have more context. Thanks,
ok, thanks any way !
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] lichaojacobs commented on pull request #2775: MAPREDUCE-7329: HadoopPipes task has failed because of the ping timeout exception
Posted by GitBox <gi...@apache.org>.
lichaojacobs commented on pull request #2775:
URL: https://github.com/apache/hadoop/pull/2775#issuecomment-808867095
@liuml07 @steveloughran hi, how is it going now? can you review this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2775: MAPREDUCE-7329: HadoopPipes task has failed because of the ping timeout exception
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2775:
URL: https://github.com/apache/hadoop/pull/2775#issuecomment-815518845
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 48s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 11s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 22m 42s | | trunk passed |
| +1 :green_heart: | compile | 2m 34s | | trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 2m 6s | | trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 56s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 10s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 42s | | trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 39s | | trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 56s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 15s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 20s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 0m 58s | | the patch passed |
| +1 :green_heart: | compile | 2m 24s | | the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 2m 24s | | the patch passed |
| +1 :green_heart: | compile | 2m 2s | | the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | javac | 2m 2s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 50s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 57s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 31s | | the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 29s | | the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 2m 7s | | the patch passed |
| +1 :green_heart: | shadedclient | 16m 19s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 7m 42s | [/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2775/10/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt) | hadoop-mapreduce-client-core in the patch passed. |
| -1 :x: | unit | 135m 17s | [/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2775/10/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt) | hadoop-mapreduce-client-jobclient in the patch passed. |
| +1 :green_heart: | asflicense | 0m 34s | | The patch does not generate ASF License warnings. |
| | | 236m 19s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.mapred.TestJobEndNotifier |
| | hadoop.mapreduce.TestMRJobClient |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2775/10/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2775 |
| JIRA Issue | MAPREDUCE-7329 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 8a43060978c9 4.15.0-126-generic #129-Ubuntu SMP Mon Nov 23 18:53:38 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 9d4443f5502d5ff82fedcdbc99571980404d1478 |
| Default Java | Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2775/10/testReport/ |
| Max. process+thread count | 1192 (vs. ulimit of 5500) |
| modules | C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2775/10/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] aajisaka commented on pull request #2775: MAPREDUCE-7329: HadoopPipes task has failed because of the ping timeout exception
Posted by GitBox <gi...@apache.org>.
aajisaka commented on pull request #2775:
URL: https://github.com/apache/hadoop/pull/2775#issuecomment-815552115
The test failures look unrelated to the patch.
- TestJobEndNotifier is failing on trunk. Filed MAPREDUCE-7334.
- TestMRJobClient seems flaky. FIled MAPREDUCE-7335
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2775: MAPREDUCE-7329: HadoopPipes task has failed because of the ping timeout exception
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2775:
URL: https://github.com/apache/hadoop/pull/2775#issuecomment-815509535
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 42s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 29s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 20m 1s | | trunk passed |
| +1 :green_heart: | compile | 2m 24s | | trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 2m 3s | | trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 58s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 17s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 53s | | trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 48s | | trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 2m 1s | | trunk passed |
| +1 :green_heart: | shadedclient | 13m 43s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 25s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 1s | | the patch passed |
| +1 :green_heart: | compile | 2m 17s | | the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 2m 17s | | the patch passed |
| +1 :green_heart: | compile | 1m 56s | | the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | javac | 1m 56s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 49s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 1s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 36s | | the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 34s | | the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 2m 6s | | the patch passed |
| +1 :green_heart: | shadedclient | 13m 39s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 7m 28s | [/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2775/9/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt) | hadoop-mapreduce-client-core in the patch passed. |
| +1 :green_heart: | unit | 129m 48s | | hadoop-mapreduce-client-jobclient in the patch passed. |
| +1 :green_heart: | asflicense | 0m 40s | | The patch does not generate ASF License warnings. |
| | | 223m 48s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.mapred.TestJobEndNotifier |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2775/9/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2775 |
| JIRA Issue | MAPREDUCE-7329 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux eb1df8066109 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 9d4443f5502d5ff82fedcdbc99571980404d1478 |
| Default Java | Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2775/9/testReport/ |
| Max. process+thread count | 1607 (vs. ulimit of 5500) |
| modules | C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2775/9/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] lichaojacobs commented on a change in pull request #2775: MAPREDUCE-7329: HadoopPipes task has failed because of the ping timeout exception
Posted by GitBox <gi...@apache.org>.
lichaojacobs commented on a change in pull request #2775:
URL: https://github.com/apache/hadoop/pull/2775#discussion_r603754126
##########
File path: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/pipes/Application.java
##########
@@ -266,4 +271,39 @@ public static String createDigest(byte[] password, String data)
return SecureShuffleUtils.hashFromString(data, key);
}
+ private class PingSocketCleaner extends Thread {
+ PingSocketCleaner(String name) {
+ super(name);
+ }
+
+ @Override
+ public void run() {
+ LOG.info("PingSocketCleaner started...");
+ while (true) {
+ Socket clientSocket = null;
+ try {
+ clientSocket = serverSocket.accept();
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("Got one client socket...");
+ }
+ int readData = clientSocket.getInputStream().read();
Review comment:
and i think we can do like this:
```
int readData = 0;
while (readData != -1) {
readData = clientSocket.getInputStream().read();
}
LOG.debug("close socket cause client has closed.");
clientSocket.close();
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] lichaojacobs commented on pull request #2775: MAPREDUCE-7329: HadoopPipes task has failed because of the ping timeout exception
Posted by GitBox <gi...@apache.org>.
lichaojacobs commented on pull request #2775:
URL: https://github.com/apache/hadoop/pull/2775#issuecomment-815512175
@aajisaka Hi, would you please help to review some changes about yesterday's advice again ? And i don't know why there is 💔 -1 overall cause i have not changed that part code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] aajisaka commented on pull request #2775: MAPREDUCE-7329: HadoopPipes task has failed because of the ping timeout exception
Posted by GitBox <gi...@apache.org>.
aajisaka commented on pull request #2775:
URL: https://github.com/apache/hadoop/pull/2775#issuecomment-814654347
Can `ServerSocket#setSoTimeout(int)` be used instead of modifying the kernel parameter?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] lichaojacobs commented on pull request #2775: MAPREDUCE-7329: HadoopPipes task has failed because of the ping timeout exception
Posted by GitBox <gi...@apache.org>.
lichaojacobs commented on pull request #2775:
URL: https://github.com/apache/hadoop/pull/2775#issuecomment-814699686
> > Of course, but need to accept client socket first?
>
> No, this method must be called before accepting client sockets.
>
> > if we set so timeout, then which value is suitable?
>
> The default value is 60 seconds in Hadoop IPC client (ipc.ping.interval). Is it too long for Hadoop Pipes application?
The value sounds reasonable. And I think we should set so timeout on client socket? cause `ServerSocket#setSoTimeout(int)` just affect `accept`
And we should do like this:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org