You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@maven.apache.org by GitBox <gi...@apache.org> on 2022/05/06 09:31:47 UTC
[GitHub] [maven-surefire] sman-81 opened a new pull request, #531: [SUREFIRE-2082] Close file handles asap to prevent breaching the system's maximum number of open files
sman-81 opened a new pull request, #531:
URL: https://github.com/apache/maven-surefire/pull/531
This pull request fixes bug SUREFIRE-2082 by closing stdout/stderr capture file handles of a huge test set as early as possible rather than at the completion of the whole set.
Following this checklist to help us incorporate your contribution quickly and easily:
- [x] Make sure there is a [JIRA issue](https://issues.apache.org/jira/browse/SUREFIRE) filed
for the change (usually before you start working on it). Trivial changes like typos do not
require a JIRA issue. Your pull request should address just this issue, without
pulling in other changes.
- [x] Each commit in the pull request should have a meaningful subject line and body.
- [x] Format the pull request title like `[SUREFIRE-XXX] - Fixes bug in ApproximateQuantiles`,
where you replace `SUREFIRE-XXX` with the appropriate JIRA issue. Best practice
is to use the JIRA issue title in the pull request title and in the first line of the
commit message.
- [x] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
- [x] Run `mvn clean install` to make sure basic checks pass. A more thorough check will
be performed on your pull request automatically.
- [ ] You have run the integration tests successfully (`mvn -Prun-its clean install`).
If your pull request is about ~20 lines of code you don't need to sign an
[Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf) if you are unsure
please ask on the developers list.
To make clear that you license your contribution under
the [Apache License Version 2.0, January 2004](http://www.apache.org/licenses/LICENSE-2.0)
you have to acknowledge this by using the following check-box.
- [x] I hereby declare this contribution to be licenced under the [Apache License Version 2.0, January 2004](http://www.apache.org/licenses/LICENSE-2.0)
- [ ] In any other case, please file an [Apache Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [maven-surefire] Tibor17 commented on pull request #531: [SUREFIRE-2082] - Close file handles asap to prevent breaching the system's maximum number of open files
Posted by GitBox <gi...@apache.org>.
Tibor17 commented on PR #531:
URL: https://github.com/apache/maven-surefire/pull/531#issuecomment-1146586403
@sman-81
Sorry that I am late. I will go ahead by today or this week. I have to talk with my colleagues.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [maven-surefire] Tibor17 commented on pull request #531: [SUREFIRE-2082] - Close file handles asap to prevent breaching the system's maximum number of open files
Posted by GitBox <gi...@apache.org>.
Tibor17 commented on PR #531:
URL: https://github.com/apache/maven-surefire/pull/531#issuecomment-1124407880
Here are my changes #534, we can talk about it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [maven-surefire] sman-81 commented on pull request #531: [SUREFIRE-2082] - Close file handles asap to prevent breaching the system's maximum number of open files
Posted by GitBox <gi...@apache.org>.
sman-81 commented on PR #531:
URL: https://github.com/apache/maven-surefire/pull/531#issuecomment-1119952649
> @sman-81 please verify IT tests result ...
wdym? `mvn -Prun-its clean install` is green locally on this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [maven-surefire] sman-81 commented on pull request #531: [SUREFIRE-2082] - Close file handles asap to prevent breaching the system's maximum number of open files
Posted by GitBox <gi...@apache.org>.
sman-81 commented on PR #531:
URL: https://github.com/apache/maven-surefire/pull/531#issuecomment-1120184263
> ... strange but build on CI are green
mysterious!
My repo is up-to-date with upstream apache:master.
Do the CI builds include integration tests?
Is the build green if you run this locally:
```
git checkout master
mvn --errors --batch-mode --show-version
-D"invoker.streamLogsOnFailures" clean install -nsu -P run-its -Dit.test=Surefire1367AssumptionLogsIT
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [maven-surefire] sman-81 commented on pull request #531: [SUREFIRE-2082] - Close file handles asap to prevent breaching the system's maximum number of open files
Posted by GitBox <gi...@apache.org>.
sman-81 commented on PR #531:
URL: https://github.com/apache/maven-surefire/pull/531#issuecomment-1125032316
> Here are my changes #534, we can talk about it.
Please feel free to go ahead and use your solution. I close my pull request.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [maven-surefire] Tibor17 commented on pull request #531: [SUREFIRE-2082] - Close file handles asap to prevent breaching the system's maximum number of open files
Posted by GitBox <gi...@apache.org>.
Tibor17 commented on PR #531:
URL: https://github.com/apache/maven-surefire/pull/531#issuecomment-1151644647
@sman-81
Thx for contributing anyway!
T
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [maven-surefire] slawekjaranowski commented on pull request #531: [SUREFIRE-2082] - Close file handles asap to prevent breaching the system's maximum number of open files
Posted by GitBox <gi...@apache.org>.
slawekjaranowski commented on PR #531:
URL: https://github.com/apache/maven-surefire/pull/531#issuecomment-1120012303
eg: https://github.com/apache/maven-surefire/actions/runs/2280904237
at the bottom of the page there are artifacts with logs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [maven-surefire] slawekjaranowski commented on pull request #531: [SUREFIRE-2082] - Close file handles asap to prevent breaching the system's maximum number of open files
Posted by GitBox <gi...@apache.org>.
slawekjaranowski commented on PR #531:
URL: https://github.com/apache/maven-surefire/pull/531#issuecomment-1120169304
... strange but build on CI are green
https://ci-maven.apache.org/job/Maven/job/maven-box/job/maven-surefire/job/master/
https://github.com/apache/maven-surefire/actions/workflows/maven-verify.yml?query=branch%3Amaster
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [maven-surefire] slawekjaranowski commented on pull request #531: [SUREFIRE-2082] - Close file handles asap to prevent breaching the system's maximum number of open files
Posted by GitBox <gi...@apache.org>.
slawekjaranowski commented on PR #531:
URL: https://github.com/apache/maven-surefire/pull/531#issuecomment-1120191197
>
> Do the CI builds include integration tests?
>
yes - with many oses
> Is the build green if you run this locally:
yes
```
Apache Maven 3.8.5 (3599d3414f046de2324203b78ddcf9b5e4388aa0)
Maven home: /usr/local/Cellar/maven/3.8.5/libexec
Java version: 1.8.0_322, vendor: Homebrew, runtime: /usr/local/Cellar/openjdk@8/1.8.0+322/libexec/openjdk.jdk/Contents/Home/jre
Default locale: en_GB, platform encoding: UTF-8
OS name: "mac os x", version: "12.3.1", arch: "x86_64", family: "mac"
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.maven.surefire.its.jiras.Surefire1367AssumptionLogsIT
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.914 s - in org.apache.maven.surefire.its.jiras.Surefire1367AssumptionLogsIT
```
```
git show -s
commit fea624d1503eadfcf95e979bdac7438db05669a5 (HEAD -> master, upstream/master, origin/master, origin/HEAD)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [maven-surefire] sman-81 commented on pull request #531: [SUREFIRE-2082] - Close file handles asap to prevent breaching the system's maximum number of open files
Posted by GitBox <gi...@apache.org>.
sman-81 commented on PR #531:
URL: https://github.com/apache/maven-surefire/pull/531#issuecomment-1120633345
> Utf8RecodingDeferredFileOutputStream.java should not be changed from Java NIO to Java IO, it cost me some time to do it with NIO.
The member `file` was back changed from `Path` to `File`, as the original code mostly operates on `File` (`Path.toFile()` called in serveral places).
> You involved two interests in one method. Write is write operation. Close is close.
The original code opens the file in `write`, it writes the file in `free`. There was no clear separation of interests before. Why should my pull request have to introduce a separation?
> I can show you my proposal to keep the file open between testStarting and test*** but then the method `writeTo` has to reopen it for reading. But is still does not make sense to do this because I am reworking it. Currently it is supposed that the test events are sequential.
wym?
The original code opens one file for stdout and one for stderr redirection for each test case (writing to stdout/err) and **keeps the file handles open until completion of the whole test set** _<--_ this is were the original code is flawed!
Example: a parameterized test with 2000 test iterations that write to stdout and stderr hogs 4000 file handles (_in addition_ to the rest of surefire and system) and will fail on Linux assuming a maximum number of file descriptors (`ulimit -n`) setting of 4096.
I've seen it happen, thus the bug report and pull request. The PR resolves the issue nicely.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [maven-surefire] Tibor17 closed pull request #531: [SUREFIRE-2082] - Close file handles asap to prevent breaching the system's maximum number of open files
Posted by GitBox <gi...@apache.org>.
Tibor17 closed pull request #531: [SUREFIRE-2082] - Close file handles asap to prevent breaching the system's maximum number of open files
URL: https://github.com/apache/maven-surefire/pull/531
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [maven-surefire] slawekjaranowski commented on pull request #531: [SUREFIRE-2082] - Close file handles asap to prevent breaching the system's maximum number of open files
Posted by GitBox <gi...@apache.org>.
slawekjaranowski commented on PR #531:
URL: https://github.com/apache/maven-surefire/pull/531#issuecomment-1119917775
@sman-81 please verify IT tests result ...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [maven-surefire] slawekjaranowski commented on pull request #531: [SUREFIRE-2082] - Close file handles asap to prevent breaching the system's maximum number of open files
Posted by GitBox <gi...@apache.org>.
slawekjaranowski commented on PR #531:
URL: https://github.com/apache/maven-surefire/pull/531#issuecomment-1120010935
Please look at result of build on GH
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [maven-surefire] sman-81 commented on pull request #531: [SUREFIRE-2082] - Close file handles asap to prevent breaching the system's maximum number of open files
Posted by GitBox <gi...@apache.org>.
sman-81 commented on PR #531:
URL: https://github.com/apache/maven-surefire/pull/531#issuecomment-1120167809
```
git checkout master
mvn --errors --batch-mode --show-version -D"invoker.streamLogsOnFailures" clean install -nsu -P run-its -Dit.test=Surefire1367AssumptionLogsIT
```
```
[ERROR] Failures:
[ERROR] Surefire1367AssumptionLogsIT.shouldSeeLogsForked:66->verifyReportA:93
[ERROR] Surefire1367AssumptionLogsIT.shouldSeeLogsInPlugin:76->verifyReportA:93
[ERROR] Surefire1367AssumptionLogsIT.shouldSeeLogsParallelForked:46->verifyReportB:109
[ERROR] Surefire1367AssumptionLogsIT.shouldSeeLogsParallelInPlugin:57->verifyReportB:109
[ERROR] Tests run: 4, Failures: 4, Errors: 0, Skipped: 0
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-failsafe-plugin:3.0.0-M6:verify (default) on project surefire-its: There are test failures.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [maven-surefire] sman-81 commented on pull request #531: [SUREFIRE-2082] - Close file handles asap to prevent breaching the system's maximum number of open files
Posted by GitBox <gi...@apache.org>.
sman-81 commented on PR #531:
URL: https://github.com/apache/maven-surefire/pull/531#issuecomment-1120130240
> eg: https://github.com/apache/maven-surefire/actions/runs/2280904237 at the bottom of the page there are artifacts with logs
After combing through the logs I found test failures in `Surefire1367AssumptionLogsIT`.
I will try to reproduce using the same command line as the Github build job:
`mvn --errors --batch-mode --show-version -D"invoker.streamLogsOnFailures" clean install -nsu -P run-its`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [maven-surefire] Tibor17 commented on pull request #531: [SUREFIRE-2082] - Close file handles asap to prevent breaching the system's maximum number of open files
Posted by GitBox <gi...@apache.org>.
Tibor17 commented on PR #531:
URL: https://github.com/apache/maven-surefire/pull/531#issuecomment-1120221689
Utf8RecodingDeferredFileOutputStream.java should not be changed from Java NIO to Java IO, it cost me some time to do it with NIO. You involved two interests in one method. Write is write operation. Close is close.
I can show you my proposal to keep the file open between testStarting and test*** but then the method `writeTo` has to reopen it for reading. But is still does not make sense to do this because I am reworking it. Currently it is supposed that the test events are sequential.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org