You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@poi.apache.org by GitBox <gi...@apache.org> on 2021/11/02 16:26:01 UTC
[GitHub] [poi] manticore-projects opened a new pull request #275: Better parallel build performance
manticore-projects opened a new pull request #275:
URL: https://github.com/apache/poi/pull/275
Following tweaks to Gradle reduces JAR build time by 27%:
1) `gradle clean jar` (after a first full warmup build), on a Ryzen7 5800U (8 physical cores, 16GB Ram)
2) the measurement is not scientific, I just used the Gradle Build time and Variable CPU frequencies and worked in parallel on the computer
3) the measurement has had around 15 seconds variance
4) original build time was around 08:40 plus/minus 15 sec. new build time reduce to 06:20 plus/minus 15 seconds
5) build still worked on a memory starved laptop (2 Cores, 3 GByte)
6) build was stable, I ran more than 30 tests with various combinations of settings. The commit shows the best/optimal combination.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For additional commands, e-mail: dev-help@poi.apache.org
[GitHub] [poi] manticore-projects commented on pull request #275: Better parallel build performance
Posted by GitBox <gi...@apache.org>.
manticore-projects commented on pull request #275:
URL: https://github.com/apache/poi/pull/275#issuecomment-958740030
I gave it my best try:
On CI please call gradle as shown below in order to achieve serial build and Testing on 2 threads
```bash
CI_BUILD=TRUE gradle --no-parallel --max-workers=1 clean build
```
On a normal build environment, just call
```
gradle clean build
```
in order to achieve parallel build.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For additional commands, e-mail: dev-help@poi.apache.org
[GitHub] [poi] manticore-projects commented on a change in pull request #275: Better parallel build performance
Posted by GitBox <gi...@apache.org>.
manticore-projects commented on a change in pull request #275:
URL: https://github.com/apache/poi/pull/275#discussion_r741661358
##########
File path: build.gradle
##########
@@ -271,8 +278,10 @@ subprojects {
'-Djavax.xml.stream.XMLInputFactory=com.sun.xml.internal.stream.XMLInputFactoryImpl',
"-Dversion.id=${project.version}",
'-ea',
- '-Djunit.jupiter.execution.parallel.config.strategy=fixed',
- '-Djunit.jupiter.execution.parallel.config.fixed.parallelism=2'
+ '-Djunit.jupiter.execution.parallel.enabled=true',
+ '-Djunit.jupiter.execution.parallel..mode.default=concurrent'
Review comment:
Would it be possible to rely on an Environment Variable, e. g.:
```
boolean isJenkinsBuild = Boolean.valueOf(System.getenv("CI_BUILD"));
boolean isSoftwarePresent = new ProcessBuilder("check software presence").start().waitFor() == 0;
Assume.assumeTrue("Software not present", isSoftwarePresent || isJenkinsBuild);
```
for restricting the build to serial mode?
Would you be able and willing to set the Environment Variable "CI_BUILD" when kicking off Jenkins:
```
pipeline {
stage('test'){
sh '''CI_BUILD="TRUE" // setting the env variable in the same shell where you are running mvn
mvn test'''
}
}
```
Any better idea?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For additional commands, e-mail: dev-help@poi.apache.org
[GitHub] [poi] centic9 commented on a change in pull request #275: Better parallel build performance
Posted by GitBox <gi...@apache.org>.
centic9 commented on a change in pull request #275:
URL: https://github.com/apache/poi/pull/275#discussion_r742018411
##########
File path: gradle.properties
##########
@@ -1,3 +1,15 @@
# Specifies the JVM arguments used for the daemon process.
# The setting is particularly useful for tweaking memory settings.
-org.gradle.jvmargs=-Xmx4096m
\ No newline at end of file
+# Less than 2G definitely slows things down.
+org.gradle.jvmargs=-Xmx2G -XX:MaxPermSize=512m -XX:+HeapDumpOnOutOfMemoryError -Dfile.encoding=UTF-8
Review comment:
Thanks, if the full build via the "jenkins" task still works it should be fine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For additional commands, e-mail: dev-help@poi.apache.org
[GitHub] [poi] manticore-projects commented on a change in pull request #275: Better parallel build performance
Posted by GitBox <gi...@apache.org>.
manticore-projects commented on a change in pull request #275:
URL: https://github.com/apache/poi/pull/275#discussion_r741655886
##########
File path: build.gradle
##########
@@ -271,8 +278,10 @@ subprojects {
'-Djavax.xml.stream.XMLInputFactory=com.sun.xml.internal.stream.XMLInputFactoryImpl',
"-Dversion.id=${project.version}",
'-ea',
- '-Djunit.jupiter.execution.parallel.config.strategy=fixed',
- '-Djunit.jupiter.execution.parallel.config.fixed.parallelism=2'
+ '-Djunit.jupiter.execution.parallel.enabled=true',
+ '-Djunit.jupiter.execution.parallel..mode.default=concurrent'
Review comment:
> So we should avoid running into the same problems again. Can we have at least a way to configure the run in CI to not run with full parallel build so we can keep CI stable without adding such hard to track-down breakages?
Thank you for the feedback. It is definitely possible, although I have one major question: How to detect the "CI" and to know that we are building on the "CI"?
I have built everything just fine on a 2 Core 3GB machine, is the "CI" anyhow worse equipped? Does the "CI" have a certain distinct hostname or configuration we could use for identifying?
##########
File path: gradle.properties
##########
@@ -1,3 +1,15 @@
# Specifies the JVM arguments used for the daemon process.
# The setting is particularly useful for tweaking memory settings.
-org.gradle.jvmargs=-Xmx4096m
\ No newline at end of file
+# Less than 2G definitely slows things down.
+org.gradle.jvmargs=-Xmx2G -XX:MaxPermSize=512m -XX:+HeapDumpOnOutOfMemoryError -Dfile.encoding=UTF-8
Review comment:
> Previously we needed more than 2g for running all the tests, but maybe memory requirements did change in the meantime?
I was able to run everything with Xmx=1GB, but it got slower. I found 2GB the optimal setting.
Also bear in mind, a lots of old Laptop Computers still are equipped with 4GB only (less 1GB for shared graphics making it 3 GB effectively). I think POI still should build on such a Laptop and the original build froze the system on my junk laptop computer.
> Enabling heap-dump always can lead to leftover files and disk filling up in CI, can we at least send it somewhere else, e.g. /tmp?
I will just remove it. This has no direct advantage. Apologies for accidentally "re-introducing" such stuff, I was not aware that there was a history on this already.
##########
File path: build.gradle
##########
@@ -271,8 +278,10 @@ subprojects {
'-Djavax.xml.stream.XMLInputFactory=com.sun.xml.internal.stream.XMLInputFactoryImpl',
"-Dversion.id=${project.version}",
'-ea',
- '-Djunit.jupiter.execution.parallel.config.strategy=fixed',
- '-Djunit.jupiter.execution.parallel.config.fixed.parallelism=2'
+ '-Djunit.jupiter.execution.parallel.enabled=true',
+ '-Djunit.jupiter.execution.parallel..mode.default=concurrent'
Review comment:
Would it be possible to rely on an Environment Variable, e. g.:
```
boolean isJenkinsBuild = Boolean.valueOf(System.getenv("CI_BUILD"));
boolean isSoftwarePresent = new ProcessBuilder("check software presence").start().waitFor() == 0;
Assume.assumeTrue("Software not present", isSoftwarePresent || isJenkinsBuild);
```
for restricting the build to serial mode?
Would you be able and willing to set the Environment Variable "CI_BUILD" when kicking off Jenkins:
```
pipeline {
stage('test'){
sh '''CI_BUILD="TRUE" // setting the env variable in the same shell where you are running mvn
mvn test'''
}
}
```
Any better idea?
##########
File path: build.gradle
##########
@@ -271,8 +278,10 @@ subprojects {
'-Djavax.xml.stream.XMLInputFactory=com.sun.xml.internal.stream.XMLInputFactoryImpl',
"-Dversion.id=${project.version}",
'-ea',
- '-Djunit.jupiter.execution.parallel.config.strategy=fixed',
- '-Djunit.jupiter.execution.parallel.config.fixed.parallelism=2'
+ '-Djunit.jupiter.execution.parallel.enabled=true',
+ '-Djunit.jupiter.execution.parallel..mode.default=concurrent'
Review comment:
I do not think, I damaged this `jenkins` task, I only let it set the global `CI_BUILD` variable (in case it has not been set on the command line).
##########
File path: build.gradle
##########
@@ -499,7 +533,9 @@ task allJavaDoc(type: Javadoc) {
}
-task jenkins(dependsOn: ['replaceVersion', subprojects.build, 'binDistZip','binDistTar','srcDistZip','srcDistTar']) {}
+task jenkins(dependsOn: ['replaceVersion', subprojects.build, 'binDistZip','binDistTar','srcDistZip','srcDistTar']) {
+ isCIBuild = true;
Review comment:
Sorry for being slow: does `isCIBuild = true` do any harm?
I could remove that easily of course, but I found it handy when `jenkins` always simulated the CI build.
But yes, adding the Environment Variables and also setting the Gradle Switches at https://github.com/apache/poi/blob/584c8c059b6b317178df635b37244c6e393d6c96/jenkins/create_jobs.groovy#L372 should do.
Can someone please try? :-D
##########
File path: build.gradle
##########
@@ -271,8 +278,10 @@ subprojects {
'-Djavax.xml.stream.XMLInputFactory=com.sun.xml.internal.stream.XMLInputFactoryImpl',
"-Dversion.id=${project.version}",
'-ea',
- '-Djunit.jupiter.execution.parallel.config.strategy=fixed',
- '-Djunit.jupiter.execution.parallel.config.fixed.parallelism=2'
+ '-Djunit.jupiter.execution.parallel.enabled=true',
+ '-Djunit.jupiter.execution.parallel..mode.default=concurrent'
Review comment:
> So we should avoid running into the same problems again. Can we have at least a way to configure the run in CI to not run with full parallel build so we can keep CI stable without adding such hard to track-down breakages?
Thank you for the feedback. It is definitely possible, although I have one major question: How to detect the "CI" and to know that we are building on the "CI"?
I have built everything just fine on a 2 Core 3GB machine, is the "CI" anyhow worse equipped? Does the "CI" have a certain distinct hostname or configuration we could use for identifying?
##########
File path: gradle.properties
##########
@@ -1,3 +1,15 @@
# Specifies the JVM arguments used for the daemon process.
# The setting is particularly useful for tweaking memory settings.
-org.gradle.jvmargs=-Xmx4096m
\ No newline at end of file
+# Less than 2G definitely slows things down.
+org.gradle.jvmargs=-Xmx2G -XX:MaxPermSize=512m -XX:+HeapDumpOnOutOfMemoryError -Dfile.encoding=UTF-8
Review comment:
> Previously we needed more than 2g for running all the tests, but maybe memory requirements did change in the meantime?
I was able to run everything with Xmx=1GB, but it got slower. I found 2GB the optimal setting.
Also bear in mind, a lots of old Laptop Computers still are equipped with 4GB only (less 1GB for shared graphics making it 3 GB effectively). I think POI still should build on such a Laptop and the original build froze the system on my junk laptop computer.
> Enabling heap-dump always can lead to leftover files and disk filling up in CI, can we at least send it somewhere else, e.g. /tmp?
I will just remove it. This has no direct advantage. Apologies for accidentally "re-introducing" such stuff, I was not aware that there was a history on this already.
##########
File path: build.gradle
##########
@@ -271,8 +278,10 @@ subprojects {
'-Djavax.xml.stream.XMLInputFactory=com.sun.xml.internal.stream.XMLInputFactoryImpl',
"-Dversion.id=${project.version}",
'-ea',
- '-Djunit.jupiter.execution.parallel.config.strategy=fixed',
- '-Djunit.jupiter.execution.parallel.config.fixed.parallelism=2'
+ '-Djunit.jupiter.execution.parallel.enabled=true',
+ '-Djunit.jupiter.execution.parallel..mode.default=concurrent'
Review comment:
Would it be possible to rely on an Environment Variable, e. g.:
```
boolean isJenkinsBuild = Boolean.valueOf(System.getenv("CI_BUILD"));
boolean isSoftwarePresent = new ProcessBuilder("check software presence").start().waitFor() == 0;
Assume.assumeTrue("Software not present", isSoftwarePresent || isJenkinsBuild);
```
for restricting the build to serial mode?
Would you be able and willing to set the Environment Variable "CI_BUILD" when kicking off Jenkins:
```
pipeline {
stage('test'){
sh '''CI_BUILD="TRUE" // setting the env variable in the same shell where you are running mvn
mvn test'''
}
}
```
Any better idea?
##########
File path: build.gradle
##########
@@ -271,8 +278,10 @@ subprojects {
'-Djavax.xml.stream.XMLInputFactory=com.sun.xml.internal.stream.XMLInputFactoryImpl',
"-Dversion.id=${project.version}",
'-ea',
- '-Djunit.jupiter.execution.parallel.config.strategy=fixed',
- '-Djunit.jupiter.execution.parallel.config.fixed.parallelism=2'
+ '-Djunit.jupiter.execution.parallel.enabled=true',
+ '-Djunit.jupiter.execution.parallel..mode.default=concurrent'
Review comment:
I do not think, I damaged this `jenkins` task, I only let it set the global `CI_BUILD` variable (in case it has not been set on the command line).
##########
File path: build.gradle
##########
@@ -499,7 +533,9 @@ task allJavaDoc(type: Javadoc) {
}
-task jenkins(dependsOn: ['replaceVersion', subprojects.build, 'binDistZip','binDistTar','srcDistZip','srcDistTar']) {}
+task jenkins(dependsOn: ['replaceVersion', subprojects.build, 'binDistZip','binDistTar','srcDistZip','srcDistTar']) {
+ isCIBuild = true;
Review comment:
Sorry for being slow: does `isCIBuild = true` do any harm?
I could remove that easily of course, but I found it handy when `jenkins` always simulated the CI build.
But yes, adding the Environment Variables and also setting the Gradle Switches at https://github.com/apache/poi/blob/584c8c059b6b317178df635b37244c6e393d6c96/jenkins/create_jobs.groovy#L372 should do.
Can someone please try? :-D
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For additional commands, e-mail: dev-help@poi.apache.org
[GitHub] [poi] manticore-projects edited a comment on pull request #275: Better parallel build performance
Posted by GitBox <gi...@apache.org>.
manticore-projects edited a comment on pull request #275:
URL: https://github.com/apache/poi/pull/275#issuecomment-958740030
I gave it my best try:
On CI please call gradle as shown below in order to achieve serial build and Testing on 2 threads. (Calling the `JENKINS` task would set CI_BUILD automatically.)
```bash
CI_BUILD=TRUE gradle --no-parallel --max-workers=1 clean build
```
On a normal build environment, just call
```
gradle clean build
```
in order to achieve parallel build.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For additional commands, e-mail: dev-help@poi.apache.org
[GitHub] [poi] centic9 commented on a change in pull request #275: Better parallel build performance
Posted by GitBox <gi...@apache.org>.
centic9 commented on a change in pull request #275:
URL: https://github.com/apache/poi/pull/275#discussion_r742111209
##########
File path: build.gradle
##########
@@ -271,8 +278,10 @@ subprojects {
'-Djavax.xml.stream.XMLInputFactory=com.sun.xml.internal.stream.XMLInputFactoryImpl',
"-Dversion.id=${project.version}",
'-ea',
- '-Djunit.jupiter.execution.parallel.config.strategy=fixed',
- '-Djunit.jupiter.execution.parallel.config.fixed.parallelism=2'
+ '-Djunit.jupiter.execution.parallel.enabled=true',
+ '-Djunit.jupiter.execution.parallel..mode.default=concurrent'
Review comment:
Sorry, wrong word "useable", I rather meant to keep the env-variable as single place which decides how to run the build.
If I run "jenkins" locally it would be good to still be able to decide if I want to get the speedup of the parallel building.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For additional commands, e-mail: dev-help@poi.apache.org
[GitHub] [poi] centic9 commented on pull request #275: Better parallel build performance
Posted by GitBox <gi...@apache.org>.
centic9 commented on pull request #275:
URL: https://github.com/apache/poi/pull/275#issuecomment-962631303
I have now applied a slightly adjusted version of these changes. Please take a look if it works for you now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For additional commands, e-mail: dev-help@poi.apache.org
[GitHub] [poi] centic9 commented on a change in pull request #275: Better parallel build performance
Posted by GitBox <gi...@apache.org>.
centic9 commented on a change in pull request #275:
URL: https://github.com/apache/poi/pull/275#discussion_r742019398
##########
File path: build.gradle
##########
@@ -271,8 +278,10 @@ subprojects {
'-Djavax.xml.stream.XMLInputFactory=com.sun.xml.internal.stream.XMLInputFactoryImpl',
"-Dversion.id=${project.version}",
'-ea',
- '-Djunit.jupiter.execution.parallel.config.strategy=fixed',
- '-Djunit.jupiter.execution.parallel.config.fixed.parallelism=2'
+ '-Djunit.jupiter.execution.parallel.enabled=true',
+ '-Djunit.jupiter.execution.parallel..mode.default=concurrent'
Review comment:
I'd rather keep the "jenkins" task useable and quick locally for developers as well so we can use it as a full local "pre-commit-check".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For additional commands, e-mail: dev-help@poi.apache.org
[GitHub] [poi] centic9 commented on a change in pull request #275: Better parallel build performance
Posted by GitBox <gi...@apache.org>.
centic9 commented on a change in pull request #275:
URL: https://github.com/apache/poi/pull/275#discussion_r741653034
##########
File path: gradle.properties
##########
@@ -1,3 +1,15 @@
# Specifies the JVM arguments used for the daemon process.
# The setting is particularly useful for tweaking memory settings.
-org.gradle.jvmargs=-Xmx4096m
\ No newline at end of file
+# Less than 2G definitely slows things down.
+org.gradle.jvmargs=-Xmx2G -XX:MaxPermSize=512m -XX:+HeapDumpOnOutOfMemoryError -Dfile.encoding=UTF-8
Review comment:
Previously we needed more than 2g for running all the tests, but maybe memory requirements did change in the meantime?
Enabling heap-dump always can lead to leftover files and disk filling up in CI, can we at least send it somewhere else, e.g. /tmp?
##########
File path: build.gradle
##########
@@ -271,8 +278,10 @@ subprojects {
'-Djavax.xml.stream.XMLInputFactory=com.sun.xml.internal.stream.XMLInputFactoryImpl',
"-Dversion.id=${project.version}",
'-ea',
- '-Djunit.jupiter.execution.parallel.config.strategy=fixed',
- '-Djunit.jupiter.execution.parallel.config.fixed.parallelism=2'
+ '-Djunit.jupiter.execution.parallel.enabled=true',
+ '-Djunit.jupiter.execution.parallel..mode.default=concurrent'
Review comment:
double dot seems incorrect
##########
File path: build.gradle
##########
@@ -499,7 +533,9 @@ task allJavaDoc(type: Javadoc) {
}
-task jenkins(dependsOn: ['replaceVersion', subprojects.build, 'binDistZip','binDistTar','srcDistZip','srcDistTar']) {}
+task jenkins(dependsOn: ['replaceVersion', subprojects.build, 'binDistZip','binDistTar','srcDistZip','srcDistTar']) {
+ isCIBuild = true;
Review comment:
I actually use the "jenkins" task also locally as a "Pre-Commit-Check", as otherwise things like "rat-check" and "javadoc" are not run and so you potentially commit changes which break the build.
We can safely set the environment-variables as part of the CI builds via our DSL-based approach, see https://github.com/apache/poi/blob/trunk/jenkins/create_jobs.groovy#L264 for the related code-based CI configuration.
##########
File path: gradle.properties
##########
@@ -1,3 +1,15 @@
# Specifies the JVM arguments used for the daemon process.
# The setting is particularly useful for tweaking memory settings.
-org.gradle.jvmargs=-Xmx4096m
\ No newline at end of file
+# Less than 2G definitely slows things down.
+org.gradle.jvmargs=-Xmx2G -XX:MaxPermSize=512m -XX:+HeapDumpOnOutOfMemoryError -Dfile.encoding=UTF-8
Review comment:
Thanks, if the full build via the "jenkins" task still works it should be fine.
##########
File path: build.gradle
##########
@@ -271,8 +278,10 @@ subprojects {
'-Djavax.xml.stream.XMLInputFactory=com.sun.xml.internal.stream.XMLInputFactoryImpl',
"-Dversion.id=${project.version}",
'-ea',
- '-Djunit.jupiter.execution.parallel.config.strategy=fixed',
- '-Djunit.jupiter.execution.parallel.config.fixed.parallelism=2'
+ '-Djunit.jupiter.execution.parallel.enabled=true',
+ '-Djunit.jupiter.execution.parallel..mode.default=concurrent'
Review comment:
I'd rather keep the "jenkins" task useable and quick locally for developers as well so we can use it as a full local "pre-commit-check".
##########
File path: build.gradle
##########
@@ -271,8 +278,10 @@ subprojects {
'-Djavax.xml.stream.XMLInputFactory=com.sun.xml.internal.stream.XMLInputFactoryImpl',
"-Dversion.id=${project.version}",
'-ea',
- '-Djunit.jupiter.execution.parallel.config.strategy=fixed',
- '-Djunit.jupiter.execution.parallel.config.fixed.parallelism=2'
+ '-Djunit.jupiter.execution.parallel.enabled=true',
+ '-Djunit.jupiter.execution.parallel..mode.default=concurrent'
Review comment:
Sorry, wrong word "useable", I rather meant to keep the env-variable as single place which decides how to run the build.
If I run "jenkins" locally it would be good to still be able to decide if I want to get the speedup of the parallel building.
##########
File path: build.gradle
##########
@@ -499,7 +533,9 @@ task allJavaDoc(type: Javadoc) {
}
-task jenkins(dependsOn: ['replaceVersion', subprojects.build, 'binDistZip','binDistTar','srcDistZip','srcDistTar']) {}
+task jenkins(dependsOn: ['replaceVersion', subprojects.build, 'binDistZip','binDistTar','srcDistZip','srcDistTar']) {
+ isCIBuild = true;
Review comment:
No, sorry for the confusion, it won't do harm except forcing "jenkins" to be slower even when run locally :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For additional commands, e-mail: dev-help@poi.apache.org
[GitHub] [poi] pjfanning commented on a change in pull request #275: Better parallel build performance
Posted by GitBox <gi...@apache.org>.
pjfanning commented on a change in pull request #275:
URL: https://github.com/apache/poi/pull/275#discussion_r741672436
##########
File path: build.gradle
##########
@@ -271,8 +278,10 @@ subprojects {
'-Djavax.xml.stream.XMLInputFactory=com.sun.xml.internal.stream.XMLInputFactoryImpl',
"-Dversion.id=${project.version}",
'-ea',
- '-Djunit.jupiter.execution.parallel.config.strategy=fixed',
- '-Djunit.jupiter.execution.parallel.config.fixed.parallelism=2'
+ '-Djunit.jupiter.execution.parallel.enabled=true',
+ '-Djunit.jupiter.execution.parallel..mode.default=concurrent'
Review comment:
The CI build uses the 'jenkins' task in build.gradle while human users would probably use 'test' task. So, we can have the build.gradle behave differently depending on the task that is called.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For additional commands, e-mail: dev-help@poi.apache.org
[GitHub] [poi] manticore-projects edited a comment on pull request #275: Better parallel build performance
Posted by GitBox <gi...@apache.org>.
manticore-projects edited a comment on pull request #275:
URL: https://github.com/apache/poi/pull/275#issuecomment-958740030
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For additional commands, e-mail: dev-help@poi.apache.org
[GitHub] [poi] manticore-projects edited a comment on pull request #275: Better parallel build performance
Posted by GitBox <gi...@apache.org>.
manticore-projects edited a comment on pull request #275:
URL: https://github.com/apache/poi/pull/275#issuecomment-958740030
I gave it my best try:
On CI please call gradle as shown below in order to achieve serial build and Testing on 2 threads. (Calling the `JENKINS` task would set CI_BUILD automatically.)
```bash
CI_BUILD=TRUE gradle --no-parallel --max-workers=1 clean build
```
On a normal build environment, just call
```
gradle clean build
```
in order to achieve parallel build.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For additional commands, e-mail: dev-help@poi.apache.org
[GitHub] [poi] manticore-projects commented on a change in pull request #275: Better parallel build performance
Posted by GitBox <gi...@apache.org>.
manticore-projects commented on a change in pull request #275:
URL: https://github.com/apache/poi/pull/275#discussion_r741655886
##########
File path: build.gradle
##########
@@ -271,8 +278,10 @@ subprojects {
'-Djavax.xml.stream.XMLInputFactory=com.sun.xml.internal.stream.XMLInputFactoryImpl',
"-Dversion.id=${project.version}",
'-ea',
- '-Djunit.jupiter.execution.parallel.config.strategy=fixed',
- '-Djunit.jupiter.execution.parallel.config.fixed.parallelism=2'
+ '-Djunit.jupiter.execution.parallel.enabled=true',
+ '-Djunit.jupiter.execution.parallel..mode.default=concurrent'
Review comment:
> So we should avoid running into the same problems again. Can we have at least a way to configure the run in CI to not run with full parallel build so we can keep CI stable without adding such hard to track-down breakages?
Thank you for the feedback. It is definitely possible, although I have one major question: How to detect the "CI" and to know that we are building on the "CI"?
I have built everything just fine on a 2 Core 3GB machine, is the "CI" anyhow worse equipped? Does the "CI" have a certain distinct hostname or configuration we could use for identifying?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For additional commands, e-mail: dev-help@poi.apache.org
[GitHub] [poi] pjfanning commented on a change in pull request #275: Better parallel build performance
Posted by GitBox <gi...@apache.org>.
pjfanning commented on a change in pull request #275:
URL: https://github.com/apache/poi/pull/275#discussion_r741672436
##########
File path: build.gradle
##########
@@ -271,8 +278,10 @@ subprojects {
'-Djavax.xml.stream.XMLInputFactory=com.sun.xml.internal.stream.XMLInputFactoryImpl',
"-Dversion.id=${project.version}",
'-ea',
- '-Djunit.jupiter.execution.parallel.config.strategy=fixed',
- '-Djunit.jupiter.execution.parallel.config.fixed.parallelism=2'
+ '-Djunit.jupiter.execution.parallel.enabled=true',
+ '-Djunit.jupiter.execution.parallel..mode.default=concurrent'
Review comment:
The CI build uses the 'jenkins' task in build.gradle while human users would probably use 'test' task. So, we can have the build.gradle behave differently depending on the task that is called.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For additional commands, e-mail: dev-help@poi.apache.org
[GitHub] [poi] centic9 commented on a change in pull request #275: Better parallel build performance
Posted by GitBox <gi...@apache.org>.
centic9 commented on a change in pull request #275:
URL: https://github.com/apache/poi/pull/275#discussion_r741653034
##########
File path: gradle.properties
##########
@@ -1,3 +1,15 @@
# Specifies the JVM arguments used for the daemon process.
# The setting is particularly useful for tweaking memory settings.
-org.gradle.jvmargs=-Xmx4096m
\ No newline at end of file
+# Less than 2G definitely slows things down.
+org.gradle.jvmargs=-Xmx2G -XX:MaxPermSize=512m -XX:+HeapDumpOnOutOfMemoryError -Dfile.encoding=UTF-8
Review comment:
Previously we needed more than 2g for running all the tests, but maybe memory requirements did change in the meantime?
Enabling heap-dump always can lead to leftover files and disk filling up in CI, can we at least send it somewhere else, e.g. /tmp?
##########
File path: build.gradle
##########
@@ -271,8 +278,10 @@ subprojects {
'-Djavax.xml.stream.XMLInputFactory=com.sun.xml.internal.stream.XMLInputFactoryImpl',
"-Dversion.id=${project.version}",
'-ea',
- '-Djunit.jupiter.execution.parallel.config.strategy=fixed',
- '-Djunit.jupiter.execution.parallel.config.fixed.parallelism=2'
+ '-Djunit.jupiter.execution.parallel.enabled=true',
+ '-Djunit.jupiter.execution.parallel..mode.default=concurrent'
Review comment:
double dot seems incorrect
##########
File path: build.gradle
##########
@@ -499,7 +533,9 @@ task allJavaDoc(type: Javadoc) {
}
-task jenkins(dependsOn: ['replaceVersion', subprojects.build, 'binDistZip','binDistTar','srcDistZip','srcDistTar']) {}
+task jenkins(dependsOn: ['replaceVersion', subprojects.build, 'binDistZip','binDistTar','srcDistZip','srcDistTar']) {
+ isCIBuild = true;
Review comment:
I actually use the "jenkins" task also locally as a "Pre-Commit-Check", as otherwise things like "rat-check" and "javadoc" are not run and so you potentially commit changes which break the build.
We can safely set the environment-variables as part of the CI builds via our DSL-based approach, see https://github.com/apache/poi/blob/trunk/jenkins/create_jobs.groovy#L264 for the related code-based CI configuration.
##########
File path: gradle.properties
##########
@@ -1,3 +1,15 @@
# Specifies the JVM arguments used for the daemon process.
# The setting is particularly useful for tweaking memory settings.
-org.gradle.jvmargs=-Xmx4096m
\ No newline at end of file
+# Less than 2G definitely slows things down.
+org.gradle.jvmargs=-Xmx2G -XX:MaxPermSize=512m -XX:+HeapDumpOnOutOfMemoryError -Dfile.encoding=UTF-8
Review comment:
Thanks, if the full build via the "jenkins" task still works it should be fine.
##########
File path: build.gradle
##########
@@ -271,8 +278,10 @@ subprojects {
'-Djavax.xml.stream.XMLInputFactory=com.sun.xml.internal.stream.XMLInputFactoryImpl',
"-Dversion.id=${project.version}",
'-ea',
- '-Djunit.jupiter.execution.parallel.config.strategy=fixed',
- '-Djunit.jupiter.execution.parallel.config.fixed.parallelism=2'
+ '-Djunit.jupiter.execution.parallel.enabled=true',
+ '-Djunit.jupiter.execution.parallel..mode.default=concurrent'
Review comment:
I'd rather keep the "jenkins" task useable and quick locally for developers as well so we can use it as a full local "pre-commit-check".
##########
File path: build.gradle
##########
@@ -271,8 +278,10 @@ subprojects {
'-Djavax.xml.stream.XMLInputFactory=com.sun.xml.internal.stream.XMLInputFactoryImpl',
"-Dversion.id=${project.version}",
'-ea',
- '-Djunit.jupiter.execution.parallel.config.strategy=fixed',
- '-Djunit.jupiter.execution.parallel.config.fixed.parallelism=2'
+ '-Djunit.jupiter.execution.parallel.enabled=true',
+ '-Djunit.jupiter.execution.parallel..mode.default=concurrent'
Review comment:
Sorry, wrong word "useable", I rather meant to keep the env-variable as single place which decides how to run the build.
If I run "jenkins" locally it would be good to still be able to decide if I want to get the speedup of the parallel building.
##########
File path: build.gradle
##########
@@ -499,7 +533,9 @@ task allJavaDoc(type: Javadoc) {
}
-task jenkins(dependsOn: ['replaceVersion', subprojects.build, 'binDistZip','binDistTar','srcDistZip','srcDistTar']) {}
+task jenkins(dependsOn: ['replaceVersion', subprojects.build, 'binDistZip','binDistTar','srcDistZip','srcDistTar']) {
+ isCIBuild = true;
Review comment:
No, sorry for the confusion, it won't do harm except forcing "jenkins" to be slower even when run locally :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For additional commands, e-mail: dev-help@poi.apache.org
[GitHub] [poi] manticore-projects commented on pull request #275: Better parallel build performance
Posted by GitBox <gi...@apache.org>.
manticore-projects commented on pull request #275:
URL: https://github.com/apache/poi/pull/275#issuecomment-962641527
Thank you so much, especially for the fine tuning.
It works well and I am glad that my stuff was partially helpful!
```
BUILD SUCCESSFUL in 6m 33s
```
Cheers!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For additional commands, e-mail: dev-help@poi.apache.org
[GitHub] [poi] centic9 commented on a change in pull request #275: Better parallel build performance
Posted by GitBox <gi...@apache.org>.
centic9 commented on a change in pull request #275:
URL: https://github.com/apache/poi/pull/275#discussion_r742112507
##########
File path: build.gradle
##########
@@ -499,7 +533,9 @@ task allJavaDoc(type: Javadoc) {
}
-task jenkins(dependsOn: ['replaceVersion', subprojects.build, 'binDistZip','binDistTar','srcDistZip','srcDistTar']) {}
+task jenkins(dependsOn: ['replaceVersion', subprojects.build, 'binDistZip','binDistTar','srcDistZip','srcDistTar']) {
+ isCIBuild = true;
Review comment:
No, sorry for the confusion, it won't do harm except forcing "jenkins" to be slower even when run locally :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For additional commands, e-mail: dev-help@poi.apache.org
[GitHub] [poi] manticore-projects commented on pull request #275: Better parallel build performance
Posted by GitBox <gi...@apache.org>.
manticore-projects commented on pull request #275:
URL: https://github.com/apache/poi/pull/275#issuecomment-958740030
I gave it my best try:
On CI please call gradle as shown below in order to achieve serial build and Testing on 2 threads
```bash
CI_BUILD=TRUE gradle --no-parallel --max-workers=1 clean build
```
On a normal build environment, just call
```
gradle clean build
```
in order to achieve parallel build.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For additional commands, e-mail: dev-help@poi.apache.org
[GitHub] [poi] centic9 commented on a change in pull request #275: Better parallel build performance
Posted by GitBox <gi...@apache.org>.
centic9 commented on a change in pull request #275:
URL: https://github.com/apache/poi/pull/275#discussion_r741653034
##########
File path: gradle.properties
##########
@@ -1,3 +1,15 @@
# Specifies the JVM arguments used for the daemon process.
# The setting is particularly useful for tweaking memory settings.
-org.gradle.jvmargs=-Xmx4096m
\ No newline at end of file
+# Less than 2G definitely slows things down.
+org.gradle.jvmargs=-Xmx2G -XX:MaxPermSize=512m -XX:+HeapDumpOnOutOfMemoryError -Dfile.encoding=UTF-8
Review comment:
Previously we needed more than 2g for running all the tests, but maybe memory requirements did change in the meantime?
Enabling heap-dump always can lead to leftover files and disk filling up in CI, can we at least send it somewhere else, e.g. /tmp?
##########
File path: build.gradle
##########
@@ -271,8 +278,10 @@ subprojects {
'-Djavax.xml.stream.XMLInputFactory=com.sun.xml.internal.stream.XMLInputFactoryImpl',
"-Dversion.id=${project.version}",
'-ea',
- '-Djunit.jupiter.execution.parallel.config.strategy=fixed',
- '-Djunit.jupiter.execution.parallel.config.fixed.parallelism=2'
+ '-Djunit.jupiter.execution.parallel.enabled=true',
+ '-Djunit.jupiter.execution.parallel..mode.default=concurrent'
Review comment:
double dot seems incorrect
##########
File path: build.gradle
##########
@@ -499,7 +533,9 @@ task allJavaDoc(type: Javadoc) {
}
-task jenkins(dependsOn: ['replaceVersion', subprojects.build, 'binDistZip','binDistTar','srcDistZip','srcDistTar']) {}
+task jenkins(dependsOn: ['replaceVersion', subprojects.build, 'binDistZip','binDistTar','srcDistZip','srcDistTar']) {
+ isCIBuild = true;
Review comment:
I actually use the "jenkins" task also locally as a "Pre-Commit-Check", as otherwise things like "rat-check" and "javadoc" are not run and so you potentially commit changes which break the build.
We can safely set the environment-variables as part of the CI builds via our DSL-based approach, see https://github.com/apache/poi/blob/trunk/jenkins/create_jobs.groovy#L264 for the related code-based CI configuration.
##########
File path: gradle.properties
##########
@@ -1,3 +1,15 @@
# Specifies the JVM arguments used for the daemon process.
# The setting is particularly useful for tweaking memory settings.
-org.gradle.jvmargs=-Xmx4096m
\ No newline at end of file
+# Less than 2G definitely slows things down.
+org.gradle.jvmargs=-Xmx2G -XX:MaxPermSize=512m -XX:+HeapDumpOnOutOfMemoryError -Dfile.encoding=UTF-8
Review comment:
Thanks, if the full build via the "jenkins" task still works it should be fine.
##########
File path: build.gradle
##########
@@ -271,8 +278,10 @@ subprojects {
'-Djavax.xml.stream.XMLInputFactory=com.sun.xml.internal.stream.XMLInputFactoryImpl',
"-Dversion.id=${project.version}",
'-ea',
- '-Djunit.jupiter.execution.parallel.config.strategy=fixed',
- '-Djunit.jupiter.execution.parallel.config.fixed.parallelism=2'
+ '-Djunit.jupiter.execution.parallel.enabled=true',
+ '-Djunit.jupiter.execution.parallel..mode.default=concurrent'
Review comment:
I'd rather keep the "jenkins" task useable and quick locally for developers as well so we can use it as a full local "pre-commit-check".
##########
File path: build.gradle
##########
@@ -271,8 +278,10 @@ subprojects {
'-Djavax.xml.stream.XMLInputFactory=com.sun.xml.internal.stream.XMLInputFactoryImpl',
"-Dversion.id=${project.version}",
'-ea',
- '-Djunit.jupiter.execution.parallel.config.strategy=fixed',
- '-Djunit.jupiter.execution.parallel.config.fixed.parallelism=2'
+ '-Djunit.jupiter.execution.parallel.enabled=true',
+ '-Djunit.jupiter.execution.parallel..mode.default=concurrent'
Review comment:
Sorry, wrong word "useable", I rather meant to keep the env-variable as single place which decides how to run the build.
If I run "jenkins" locally it would be good to still be able to decide if I want to get the speedup of the parallel building.
##########
File path: build.gradle
##########
@@ -499,7 +533,9 @@ task allJavaDoc(type: Javadoc) {
}
-task jenkins(dependsOn: ['replaceVersion', subprojects.build, 'binDistZip','binDistTar','srcDistZip','srcDistTar']) {}
+task jenkins(dependsOn: ['replaceVersion', subprojects.build, 'binDistZip','binDistTar','srcDistZip','srcDistTar']) {
+ isCIBuild = true;
Review comment:
No, sorry for the confusion, it won't do harm except forcing "jenkins" to be slower even when run locally :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For additional commands, e-mail: dev-help@poi.apache.org
[GitHub] [poi] pjfanning commented on a change in pull request #275: Better parallel build performance
Posted by GitBox <gi...@apache.org>.
pjfanning commented on a change in pull request #275:
URL: https://github.com/apache/poi/pull/275#discussion_r741672436
##########
File path: build.gradle
##########
@@ -271,8 +278,10 @@ subprojects {
'-Djavax.xml.stream.XMLInputFactory=com.sun.xml.internal.stream.XMLInputFactoryImpl',
"-Dversion.id=${project.version}",
'-ea',
- '-Djunit.jupiter.execution.parallel.config.strategy=fixed',
- '-Djunit.jupiter.execution.parallel.config.fixed.parallelism=2'
+ '-Djunit.jupiter.execution.parallel.enabled=true',
+ '-Djunit.jupiter.execution.parallel..mode.default=concurrent'
Review comment:
The CI build uses the 'jenkins' task in build.gradle while human users would probably use 'test' task. So, we can have the build.gradle behave differently depending on the task that is called.
##########
File path: build.gradle
##########
@@ -271,8 +278,10 @@ subprojects {
'-Djavax.xml.stream.XMLInputFactory=com.sun.xml.internal.stream.XMLInputFactoryImpl',
"-Dversion.id=${project.version}",
'-ea',
- '-Djunit.jupiter.execution.parallel.config.strategy=fixed',
- '-Djunit.jupiter.execution.parallel.config.fixed.parallelism=2'
+ '-Djunit.jupiter.execution.parallel.enabled=true',
+ '-Djunit.jupiter.execution.parallel..mode.default=concurrent'
Review comment:
The CI build uses the 'jenkins' task in build.gradle while human users would probably use 'test' task. So, we can have the build.gradle behave differently depending on the task that is called.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For additional commands, e-mail: dev-help@poi.apache.org
[GitHub] [poi] asfgit closed pull request #275: Better parallel build performance
Posted by GitBox <gi...@apache.org>.
asfgit closed pull request #275:
URL: https://github.com/apache/poi/pull/275
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For additional commands, e-mail: dev-help@poi.apache.org
[GitHub] [poi] manticore-projects commented on a change in pull request #275: Better parallel build performance
Posted by GitBox <gi...@apache.org>.
manticore-projects commented on a change in pull request #275:
URL: https://github.com/apache/poi/pull/275#discussion_r741655886
##########
File path: build.gradle
##########
@@ -271,8 +278,10 @@ subprojects {
'-Djavax.xml.stream.XMLInputFactory=com.sun.xml.internal.stream.XMLInputFactoryImpl',
"-Dversion.id=${project.version}",
'-ea',
- '-Djunit.jupiter.execution.parallel.config.strategy=fixed',
- '-Djunit.jupiter.execution.parallel.config.fixed.parallelism=2'
+ '-Djunit.jupiter.execution.parallel.enabled=true',
+ '-Djunit.jupiter.execution.parallel..mode.default=concurrent'
Review comment:
> So we should avoid running into the same problems again. Can we have at least a way to configure the run in CI to not run with full parallel build so we can keep CI stable without adding such hard to track-down breakages?
Thank you for the feedback. It is definitely possible, although I have one major question: How to detect the "CI" and to know that we are building on the "CI"?
I have built everything just fine on a 2 Core 3GB machine, is the "CI" anyhow worse equipped? Does the "CI" have a certain distinct hostname or configuration we could use for identifying?
##########
File path: gradle.properties
##########
@@ -1,3 +1,15 @@
# Specifies the JVM arguments used for the daemon process.
# The setting is particularly useful for tweaking memory settings.
-org.gradle.jvmargs=-Xmx4096m
\ No newline at end of file
+# Less than 2G definitely slows things down.
+org.gradle.jvmargs=-Xmx2G -XX:MaxPermSize=512m -XX:+HeapDumpOnOutOfMemoryError -Dfile.encoding=UTF-8
Review comment:
> Previously we needed more than 2g for running all the tests, but maybe memory requirements did change in the meantime?
I was able to run everything with Xmx=1GB, but it got slower. I found 2GB the optimal setting.
Also bear in mind, a lots of old Laptop Computers still are equipped with 4GB only (less 1GB for shared graphics making it 3 GB effectively). I think POI still should build on such a Laptop and the original build froze the system on my junk laptop computer.
> Enabling heap-dump always can lead to leftover files and disk filling up in CI, can we at least send it somewhere else, e.g. /tmp?
I will just remove it. This has no direct advantage. Apologies for accidentally "re-introducing" such stuff, I was not aware that there was a history on this already.
##########
File path: build.gradle
##########
@@ -271,8 +278,10 @@ subprojects {
'-Djavax.xml.stream.XMLInputFactory=com.sun.xml.internal.stream.XMLInputFactoryImpl',
"-Dversion.id=${project.version}",
'-ea',
- '-Djunit.jupiter.execution.parallel.config.strategy=fixed',
- '-Djunit.jupiter.execution.parallel.config.fixed.parallelism=2'
+ '-Djunit.jupiter.execution.parallel.enabled=true',
+ '-Djunit.jupiter.execution.parallel..mode.default=concurrent'
Review comment:
Would it be possible to rely on an Environment Variable, e. g.:
```
boolean isJenkinsBuild = Boolean.valueOf(System.getenv("CI_BUILD"));
boolean isSoftwarePresent = new ProcessBuilder("check software presence").start().waitFor() == 0;
Assume.assumeTrue("Software not present", isSoftwarePresent || isJenkinsBuild);
```
for restricting the build to serial mode?
Would you be able and willing to set the Environment Variable "CI_BUILD" when kicking off Jenkins:
```
pipeline {
stage('test'){
sh '''CI_BUILD="TRUE" // setting the env variable in the same shell where you are running mvn
mvn test'''
}
}
```
Any better idea?
##########
File path: build.gradle
##########
@@ -271,8 +278,10 @@ subprojects {
'-Djavax.xml.stream.XMLInputFactory=com.sun.xml.internal.stream.XMLInputFactoryImpl',
"-Dversion.id=${project.version}",
'-ea',
- '-Djunit.jupiter.execution.parallel.config.strategy=fixed',
- '-Djunit.jupiter.execution.parallel.config.fixed.parallelism=2'
+ '-Djunit.jupiter.execution.parallel.enabled=true',
+ '-Djunit.jupiter.execution.parallel..mode.default=concurrent'
Review comment:
I do not think, I damaged this `jenkins` task, I only let it set the global `CI_BUILD` variable (in case it has not been set on the command line).
##########
File path: build.gradle
##########
@@ -499,7 +533,9 @@ task allJavaDoc(type: Javadoc) {
}
-task jenkins(dependsOn: ['replaceVersion', subprojects.build, 'binDistZip','binDistTar','srcDistZip','srcDistTar']) {}
+task jenkins(dependsOn: ['replaceVersion', subprojects.build, 'binDistZip','binDistTar','srcDistZip','srcDistTar']) {
+ isCIBuild = true;
Review comment:
Sorry for being slow: does `isCIBuild = true` do any harm?
I could remove that easily of course, but I found it handy when `jenkins` always simulated the CI build.
But yes, adding the Environment Variables and also setting the Gradle Switches at https://github.com/apache/poi/blob/584c8c059b6b317178df635b37244c6e393d6c96/jenkins/create_jobs.groovy#L372 should do.
Can someone please try? :-D
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For additional commands, e-mail: dev-help@poi.apache.org
[GitHub] [poi] centic9 commented on a change in pull request #275: Better parallel build performance
Posted by GitBox <gi...@apache.org>.
centic9 commented on a change in pull request #275:
URL: https://github.com/apache/poi/pull/275#discussion_r741653034
##########
File path: gradle.properties
##########
@@ -1,3 +1,15 @@
# Specifies the JVM arguments used for the daemon process.
# The setting is particularly useful for tweaking memory settings.
-org.gradle.jvmargs=-Xmx4096m
\ No newline at end of file
+# Less than 2G definitely slows things down.
+org.gradle.jvmargs=-Xmx2G -XX:MaxPermSize=512m -XX:+HeapDumpOnOutOfMemoryError -Dfile.encoding=UTF-8
Review comment:
Previously we needed more than 2g for running all the tests, but maybe memory requirements did change in the meantime?
Enabling heap-dump always can lead to leftover files and disk filling up in CI, can we at least send it somewhere else, e.g. /tmp?
##########
File path: build.gradle
##########
@@ -271,8 +278,10 @@ subprojects {
'-Djavax.xml.stream.XMLInputFactory=com.sun.xml.internal.stream.XMLInputFactoryImpl',
"-Dversion.id=${project.version}",
'-ea',
- '-Djunit.jupiter.execution.parallel.config.strategy=fixed',
- '-Djunit.jupiter.execution.parallel.config.fixed.parallelism=2'
+ '-Djunit.jupiter.execution.parallel.enabled=true',
+ '-Djunit.jupiter.execution.parallel..mode.default=concurrent'
Review comment:
double dot seems incorrect
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For additional commands, e-mail: dev-help@poi.apache.org
[GitHub] [poi] manticore-projects commented on a change in pull request #275: Better parallel build performance
Posted by GitBox <gi...@apache.org>.
manticore-projects commented on a change in pull request #275:
URL: https://github.com/apache/poi/pull/275#discussion_r742074885
##########
File path: build.gradle
##########
@@ -499,7 +533,9 @@ task allJavaDoc(type: Javadoc) {
}
-task jenkins(dependsOn: ['replaceVersion', subprojects.build, 'binDistZip','binDistTar','srcDistZip','srcDistTar']) {}
+task jenkins(dependsOn: ['replaceVersion', subprojects.build, 'binDistZip','binDistTar','srcDistZip','srcDistTar']) {
+ isCIBuild = true;
Review comment:
Sorry for being slow: does `isCIBuild = true` do any harm?
I could remove that easily of course, but I found it handy when `jenkins` always simulated the CI build.
But yes, adding the Environment Variables and also setting the Gradle Switches at https://github.com/apache/poi/blob/584c8c059b6b317178df635b37244c6e393d6c96/jenkins/create_jobs.groovy#L372 should do.
Can someone please try? :-D
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For additional commands, e-mail: dev-help@poi.apache.org
[GitHub] [poi] centic9 commented on a change in pull request #275: Better parallel build performance
Posted by GitBox <gi...@apache.org>.
centic9 commented on a change in pull request #275:
URL: https://github.com/apache/poi/pull/275#discussion_r742015683
##########
File path: build.gradle
##########
@@ -499,7 +533,9 @@ task allJavaDoc(type: Javadoc) {
}
-task jenkins(dependsOn: ['replaceVersion', subprojects.build, 'binDistZip','binDistTar','srcDistZip','srcDistTar']) {}
+task jenkins(dependsOn: ['replaceVersion', subprojects.build, 'binDistZip','binDistTar','srcDistZip','srcDistTar']) {
+ isCIBuild = true;
Review comment:
I actually use the "jenkins" task also locally as a "Pre-Commit-Check", as otherwise things like "rat-check" and "javadoc" are not run and so you potentially commit changes which break the build.
We can safely set the environment-variables as part of the CI builds via our DSL-based approach, see https://github.com/apache/poi/blob/trunk/jenkins/create_jobs.groovy#L264 for the related code-based CI configuration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For additional commands, e-mail: dev-help@poi.apache.org
[GitHub] [poi] manticore-projects commented on pull request #275: Better parallel build performance
Posted by GitBox <gi...@apache.org>.
manticore-projects commented on pull request #275:
URL: https://github.com/apache/poi/pull/275#issuecomment-958740030
I gave it my best try:
On CI please call gradle as shown below in order to achieve serial build and Testing on 2 threads
```bash
CI_BUILD=TRUE gradle --no-parallel --max-workers=1 clean build
```
On a normal build environment, just call
```
gradle clean build
```
in order to achieve parallel build.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For additional commands, e-mail: dev-help@poi.apache.org
[GitHub] [poi] manticore-projects commented on a change in pull request #275: Better parallel build performance
Posted by GitBox <gi...@apache.org>.
manticore-projects commented on a change in pull request #275:
URL: https://github.com/apache/poi/pull/275#discussion_r742069929
##########
File path: build.gradle
##########
@@ -271,8 +278,10 @@ subprojects {
'-Djavax.xml.stream.XMLInputFactory=com.sun.xml.internal.stream.XMLInputFactoryImpl',
"-Dversion.id=${project.version}",
'-ea',
- '-Djunit.jupiter.execution.parallel.config.strategy=fixed',
- '-Djunit.jupiter.execution.parallel.config.fixed.parallelism=2'
+ '-Djunit.jupiter.execution.parallel.enabled=true',
+ '-Djunit.jupiter.execution.parallel..mode.default=concurrent'
Review comment:
I do not think, I damaged this `jenkins` task, I only let it set the global `CI_BUILD` variable (in case it has not been set on the command line).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For additional commands, e-mail: dev-help@poi.apache.org
[GitHub] [poi] manticore-projects commented on pull request #275: Better parallel build performance
Posted by GitBox <gi...@apache.org>.
manticore-projects commented on pull request #275:
URL: https://github.com/apache/poi/pull/275#issuecomment-958740030
I gave it my best try:
On CI please call gradle as shown below in order to achieve serial build and Testing on 2 threads
```bash
CI_BUILD=TRUE gradle --no-parallel --max-workers=1 clean build
```
On a normal build environment, just call
```
gradle clean build
```
in order to achieve parallel build.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For additional commands, e-mail: dev-help@poi.apache.org