You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2021/02/15 06:33:27 UTC
[GitHub] [hadoop] mpryahin opened a new pull request #2701: HADOOP-17528. Fix closing an underlying connection pool when closing SFTP File System
mpryahin opened a new pull request #2701:
URL: https://github.com/apache/hadoop/pull/2701
`SFTPFileSystem` leverages a connection pool which is not closed when a file system instance gets closed preventing a JVM from exiting as every established SFTP connection runs in a separate non-daemon thread. This PR fixes `SFTPFileSystem` closure.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran merged pull request #2701: HADOOP-17528. SFTP File System: close the connection pool when closing a FileSystem
Posted by GitBox <gi...@apache.org>.
steveloughran merged pull request #2701:
URL: https://github.com/apache/hadoop/pull/2701
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2701: HADOOP-17528. Fix closing an underlying connection pool when closing SFTP File System
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2701:
URL: https://github.com/apache/hadoop/pull/2701#issuecomment-779088595
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 30s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 9s | | trunk passed |
| +1 :green_heart: | compile | 20m 28s | | trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 17m 45s | | trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 6s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 31s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 1s | | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 5s | | trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 36s | | trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +0 :ok: | spotbugs | 2m 21s | | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 2m 19s | | trunk passed |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 54s | | the patch passed |
| +1 :green_heart: | compile | 20m 33s | | the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 20m 33s | | the patch passed |
| +1 :green_heart: | compile | 17m 53s | | the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | javac | 17m 53s | | the patch passed |
| +1 :green_heart: | checkstyle | 1m 3s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 28s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 13m 27s | | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 4s | | the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 30s | | the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | findbugs | 2m 22s | | the patch passed |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 17m 18s | | hadoop-common in the patch passed. |
| +1 :green_heart: | asflicense | 0m 50s | | The patch does not generate ASF License warnings. |
| | | 175m 5s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2701/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2701 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 937d084d102c 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / c3134ab3a99 |
| Default Java | Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2701/1/testReport/ |
| Max. process+thread count | 2197 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2701/1/console |
| versions | git=2.25.1 maven=3.6.3 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2701: HADOOP-17528. Fix closing an underlying connection pool when closing SFTP File System
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2701:
URL: https://github.com/apache/hadoop/pull/2701#issuecomment-780821638
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 37s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 50s | | trunk passed |
| +1 :green_heart: | compile | 20m 35s | | trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 17m 52s | | trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 5s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 29s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 5s | | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 4s | | trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 35s | | trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +0 :ok: | spotbugs | 21m 3s | | Both FindBugs and SpotBugs are enabled, using SpotBugs. |
| +1 :green_heart: | spotbugs | 2m 18s | | trunk passed |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 51s | | the patch passed |
| +1 :green_heart: | compile | 19m 52s | | the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 19m 52s | | the patch passed |
| +1 :green_heart: | compile | 17m 49s | | the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | javac | 17m 49s | | the patch passed |
| +1 :green_heart: | checkstyle | 1m 5s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 31s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 13m 1s | | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 6s | | the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 39s | | the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| -1 :x: | spotbugs | 2m 18s | [/patch-spotbugs-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2701/3/artifact/out/patch-spotbugs-hadoop-common-project_hadoop-common.txt) | hadoop-common-project/hadoop-common cannot run computeBugHistory from spotbugs |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 17m 22s | | hadoop-common in the patch passed. |
| +1 :green_heart: | asflicense | 0m 56s | | The patch does not generate ASF License warnings. |
| | | 175m 27s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2701/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2701 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle |
| uname | Linux 1b80d25539dd 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 0ed9e36282c |
| Default Java | Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2701/3/testReport/ |
| Max. process+thread count | 3158 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2701/3/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #2701: HADOOP-17528. Fix closing an underlying connection pool when closing SFTP File System
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #2701:
URL: https://github.com/apache/hadoop/pull/2701#discussion_r577527796
##########
File path: hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/sftp/TestSFTPFileSystem.java
##########
@@ -374,4 +374,13 @@ public void testMkDirs() throws IOException {
assertThat(((SFTPFileSystem) sftpFs).getConnectionPool().getLiveConnCount(),
is(1));
}
+
+ @Test
+ public void testCloseFileSystemClosesConnectionPool() throws Exception {
+ SFTPFileSystem fs = (SFTPFileSystem) sftpFs;
+ fs.getHomeDirectory();
+ assertThat(fs.getConnectionPool().getLiveConnCount(), is(1));
+ fs.close();
+ assertThat(fs.getConnectionPool().getLiveConnCount(), is(0));
+ }
Review comment:
we swallow those exceptions in teardown as otherwise their exceptions get reported over something in the test case itself (or setup). Please add the extra call for completeness. thanks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] mpryahin commented on pull request #2701: HADOOP-17528. Fix closing an underlying connection pool when closing SFTP File System
Posted by GitBox <gi...@apache.org>.
mpryahin commented on pull request #2701:
URL: https://github.com/apache/hadoop/pull/2701#issuecomment-781089884
@steveloughran
The build reports a problem due to the [spotbugs issue](https://github.com/spotbugs/spotbugs/issues/1161)
Here's a stacktrace from the build logs:
```
[2021-02-17T19:50:51.359Z] Building spotbugs database(s) using /opt/spotbugs for executables.
[2021-02-17T19:50:51.359Z]
[2021-02-17T19:50:51.359Z]
[2021-02-17T19:50:51.359Z] Starting with /home/jenkins/jenkins-agent/workspace/hadoop-multibranch_PR-2701/out/branch-spotbugs-hadoop-common-project_hadoop-common-warnings.xml
[2021-02-17T19:50:51.359Z] Exception in thread "main" java.lang.UnsupportedOperationException
[2021-02-17T19:50:51.359Z] at java.util.Collections$UnmodifiableCollection.removeIf(Collections.java:1086)
[2021-02-17T19:50:51.359Z] at edu.umd.cs.findbugs.BugRanker.trimToMaxRank(BugRanker.java:322)
[2021-02-17T19:50:51.359Z] at edu.umd.cs.findbugs.workflow.Update.discardUnwantedBugs(Update.java:337)
[2021-02-17T19:50:51.359Z] at edu.umd.cs.findbugs.workflow.Update.doit(Update.java:484)
[2021-02-17T19:50:51.359Z] at edu.umd.cs.findbugs.workflow.Update.main(Update.java:405)
[2021-02-17T19:50:51.359Z]
[2021-02-17T19:50:51.359Z] hadoop-common-project/hadoop-common cannot run computeBugHistory from spotbugs
```
Here's a [cause](https://github.com/spotbugs/spotbugs/blob/c7ead7e64382673b67e5b46a83442c846483230d/findbugs/src/java/edu/umd/cs/findbugs/SortedBugCollection.java#L1028) for this to happen.
This happened straight after substituting findbugs with spotbugs [here](https://github.com/apache/hadoop/commit/4cf35315838a6e65f87ed64aaa8f1d31594c7fcd)
@aajisaka could please help to fix the build?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] mpryahin commented on pull request #2701: HADOOP-17528. Fix closing an underlying connection pool when closing SFTP File System
Posted by GitBox <gi...@apache.org>.
mpryahin commented on pull request #2701:
URL: https://github.com/apache/hadoop/pull/2701#issuecomment-781970947
@aajisaka thanks you so much.
@steveloughran Steve, looking forward to getting the changes in, thank you.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #2701: HADOOP-17528. Fix closing an underlying connection pool when closing SFTP File System
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #2701:
URL: https://github.com/apache/hadoop/pull/2701#discussion_r577089216
##########
File path: hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/sftp/TestSFTPFileSystem.java
##########
@@ -374,4 +374,13 @@ public void testMkDirs() throws IOException {
assertThat(((SFTPFileSystem) sftpFs).getConnectionPool().getLiveConnCount(),
is(1));
}
+
+ @Test
+ public void testCloseFileSystemClosesConnectionPool() throws Exception {
+ SFTPFileSystem fs = (SFTPFileSystem) sftpFs;
+ fs.getHomeDirectory();
+ assertThat(fs.getConnectionPool().getLiveConnCount(), is(1));
+ fs.close();
+ assertThat(fs.getConnectionPool().getLiveConnCount(), is(0));
+ }
Review comment:
can you add another close() here just so we can make sure that re-entrant close calls are safe.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] mpryahin edited a comment on pull request #2701: HADOOP-17528. Fix closing an underlying connection pool when closing SFTP File System
Posted by GitBox <gi...@apache.org>.
mpryahin edited a comment on pull request #2701:
URL: https://github.com/apache/hadoop/pull/2701#issuecomment-781089884
@steveloughran
The build reports a problem due to the [spotbugs issue](https://github.com/spotbugs/spotbugs/issues/1161)
Here's a stacktrace from the build logs:
```
[2021-02-17T19:50:51.359Z] Building spotbugs database(s) using /opt/spotbugs for executables.
[2021-02-17T19:50:51.359Z]
[2021-02-17T19:50:51.359Z]
[2021-02-17T19:50:51.359Z] Starting with /home/jenkins/jenkins-agent/workspace/hadoop-multibranch_PR-2701/out/branch-spotbugs-hadoop-common-project_hadoop-common-warnings.xml
[2021-02-17T19:50:51.359Z] Exception in thread "main" java.lang.UnsupportedOperationException
[2021-02-17T19:50:51.359Z] at java.util.Collections$UnmodifiableCollection.removeIf(Collections.java:1086)
[2021-02-17T19:50:51.359Z] at edu.umd.cs.findbugs.BugRanker.trimToMaxRank(BugRanker.java:322)
[2021-02-17T19:50:51.359Z] at edu.umd.cs.findbugs.workflow.Update.discardUnwantedBugs(Update.java:337)
[2021-02-17T19:50:51.359Z] at edu.umd.cs.findbugs.workflow.Update.doit(Update.java:484)
[2021-02-17T19:50:51.359Z] at edu.umd.cs.findbugs.workflow.Update.main(Update.java:405)
[2021-02-17T19:50:51.359Z]
[2021-02-17T19:50:51.359Z] hadoop-common-project/hadoop-common cannot run computeBugHistory from spotbugs
```
Here's a [cause](https://github.com/spotbugs/spotbugs/blob/c7ead7e64382673b67e5b46a83442c846483230d/findbugs/src/java/edu/umd/cs/findbugs/SortedBugCollection.java#L1028) for this to happen.
This happened straight after substituting findbugs with spotbugs [here](https://github.com/apache/hadoop/commit/4cf35315838a6e65f87ed64aaa8f1d31594c7fcd)
@aajisaka could you please help to fix the build?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #2701: HADOOP-17528. SFTP File System: close the connection pool when closing a FileSystem
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #2701:
URL: https://github.com/apache/hadoop/pull/2701#issuecomment-784353570
+1, merged to trunk
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] mpryahin commented on a change in pull request #2701: HADOOP-17528. Fix closing an underlying connection pool when closing SFTP File System
Posted by GitBox <gi...@apache.org>.
mpryahin commented on a change in pull request #2701:
URL: https://github.com/apache/hadoop/pull/2701#discussion_r577531913
##########
File path: hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/sftp/TestSFTPFileSystem.java
##########
@@ -374,4 +374,13 @@ public void testMkDirs() throws IOException {
assertThat(((SFTPFileSystem) sftpFs).getConnectionPool().getLiveConnCount(),
is(1));
}
+
+ @Test
+ public void testCloseFileSystemClosesConnectionPool() throws Exception {
+ SFTPFileSystem fs = (SFTPFileSystem) sftpFs;
+ fs.getHomeDirectory();
+ assertThat(fs.getConnectionPool().getLiveConnCount(), is(1));
+ fs.close();
+ assertThat(fs.getConnectionPool().getLiveConnCount(), is(0));
+ }
Review comment:
exactly, I missed the catch block, thank you! Done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2701: HADOOP-17528. Fix closing an underlying connection pool when closing SFTP File System
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2701:
URL: https://github.com/apache/hadoop/pull/2701#issuecomment-781961262
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 30s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 56s | | trunk passed |
| +1 :green_heart: | compile | 23m 27s | | trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 22m 29s | | trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 13s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 44s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 34s | | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 8s | | trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 48s | | trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +0 :ok: | spotbugs | 3m 0s | | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 2m 56s | | trunk passed |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 7s | | the patch passed |
| +1 :green_heart: | compile | 26m 48s | | the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 26m 48s | | the patch passed |
| +1 :green_heart: | compile | 23m 11s | | the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | javac | 23m 11s | | the patch passed |
| +1 :green_heart: | checkstyle | 1m 12s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 44s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 17m 19s | | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 6s | | the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 47s | | the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | findbugs | 2m 57s | | the patch passed |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 18m 38s | | hadoop-common in the patch passed. |
| +1 :green_heart: | asflicense | 0m 55s | | The patch does not generate ASF License warnings. |
| | | 207m 36s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2701/5/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2701 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux ec4ed8bbbc80 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 25af8901a97 |
| Default Java | Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2701/5/testReport/ |
| Max. process+thread count | 1389 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2701/5/console |
| versions | git=2.25.1 maven=3.6.3 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] aajisaka commented on pull request #2701: HADOOP-17528. Fix closing an underlying connection pool when closing SFTP File System
Posted by GitBox <gi...@apache.org>.
aajisaka commented on pull request #2701:
URL: https://github.com/apache/hadoop/pull/2701#issuecomment-781765147
Reverted #2454
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] aajisaka commented on pull request #2701: HADOOP-17528. Fix closing an underlying connection pool when closing SFTP File System
Posted by GitBox <gi...@apache.org>.
aajisaka commented on pull request #2701:
URL: https://github.com/apache/hadoop/pull/2701#issuecomment-781763970
Thank you @mpryahin for your report. I'll revert #2454 shortly.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] mpryahin commented on a change in pull request #2701: HADOOP-17528. Fix closing an underlying connection pool when closing SFTP File System
Posted by GitBox <gi...@apache.org>.
mpryahin commented on a change in pull request #2701:
URL: https://github.com/apache/hadoop/pull/2701#discussion_r577357676
##########
File path: hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/sftp/TestSFTPFileSystem.java
##########
@@ -374,4 +374,13 @@ public void testMkDirs() throws IOException {
assertThat(((SFTPFileSystem) sftpFs).getConnectionPool().getLiveConnCount(),
is(1));
}
+
+ @Test
+ public void testCloseFileSystemClosesConnectionPool() throws Exception {
+ SFTPFileSystem fs = (SFTPFileSystem) sftpFs;
+ fs.getHomeDirectory();
+ assertThat(fs.getConnectionPool().getLiveConnCount(), is(1));
+ fs.close();
+ assertThat(fs.getConnectionPool().getLiveConnCount(), is(0));
+ }
Review comment:
Hey Steve, thanks a lot for reviewing, I appreciate it.
A FileSystem instance already gets closed after each test method, [please take a look over here](https://github.com/apache/hadoop/blob/ca2e5a462daeb48f2e18e6a0ebe2fd6a3f20e89e/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/sftp/TestSFTPFileSystem.java#L117). I hope this makes the case you pointed out covered. Please, let me know if it doesn't, I will add an extra call to make it look more explicit.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2701: HADOOP-17528. Fix closing an underlying connection pool when closing SFTP File System
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2701:
URL: https://github.com/apache/hadoop/pull/2701#issuecomment-780594306
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 14m 18s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 27s | | trunk passed |
| +1 :green_heart: | compile | 20m 30s | | trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 17m 48s | | trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 5s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 30s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 0s | | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 4s | | trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 37s | | trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +0 :ok: | spotbugs | 21m 5s | | Both FindBugs and SpotBugs are enabled, using SpotBugs. |
| +1 :green_heart: | spotbugs | 2m 23s | | trunk passed |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 54s | | the patch passed |
| +1 :green_heart: | compile | 19m 53s | | the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 19m 53s | | the patch passed |
| +1 :green_heart: | compile | 17m 51s | | the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | javac | 17m 51s | | the patch passed |
| +1 :green_heart: | checkstyle | 1m 4s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 29s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 13m 17s | | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 1s | | the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 37s | | the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| -1 :x: | spotbugs | 2m 17s | [/patch-spotbugs-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2701/2/artifact/out/patch-spotbugs-hadoop-common-project_hadoop-common.txt) | hadoop-common-project/hadoop-common cannot run computeBugHistory from spotbugs |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 17m 9s | | hadoop-common in the patch passed. |
| +1 :green_heart: | asflicense | 0m 55s | | The patch does not generate ASF License warnings. |
| | | 188m 30s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2701/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2701 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle |
| uname | Linux 44c4dabf0b93 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 0ed9e36282c |
| Default Java | Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2701/2/testReport/ |
| Max. process+thread count | 1853 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2701/2/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2701: HADOOP-17528. Fix closing an underlying connection pool when closing SFTP File System
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2701:
URL: https://github.com/apache/hadoop/pull/2701#issuecomment-781907524
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 15m 19s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 35m 59s | | trunk passed |
| +1 :green_heart: | compile | 23m 10s | | trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 20m 23s | | trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 5s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 33s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 21s | | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 3s | | trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 33s | | trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +0 :ok: | spotbugs | 2m 23s | | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 2m 21s | | trunk passed |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 57s | | the patch passed |
| +1 :green_heart: | compile | 21m 36s | | the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 21m 36s | | the patch passed |
| +1 :green_heart: | compile | 19m 35s | | the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | javac | 19m 35s | | the patch passed |
| +1 :green_heart: | checkstyle | 1m 2s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 23s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 13m 7s | | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 4s | | the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 35s | | the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | findbugs | 2m 38s | | the patch passed |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 17m 41s | | hadoop-common in the patch passed. |
| +1 :green_heart: | asflicense | 0m 53s | | The patch does not generate ASF License warnings. |
| | | 202m 37s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2701/4/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2701 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux bf626a3dc6fd 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 9a298d180da |
| Default Java | Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2701/4/testReport/ |
| Max. process+thread count | 3158 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2701/4/console |
| versions | git=2.25.1 maven=3.6.3 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org