You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@jena.apache.org by rvesse <gi...@git.apache.org> on 2017/10/26 17:08:11 UTC
[GitHub] jena pull request #297: Enable parallel testing for Elephas IO (JENA-1407)
GitHub user rvesse opened a pull request:
https://github.com/apache/jena/pull/297
Enable parallel testing for Elephas IO (JENA-1407)
Using parallelism of 4 reduces build time for Elephas IO to
approximately 3-4 minutes in my testing. Also fixed one test that was
not parallel safe to use the pre-existing temporary folder.
Ran this a ton of times in a tight loop on my system to check that there weren't any transient errors or serial test assumptions encoded in the tests.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/rvesse/jena JENA-1407
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/jena/pull/297.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #297
----
commit b26643b4cf19704cd4ad969d65f59095b235d667
Author: Rob Vesse <rv...@apache.org>
Date: 2017-10-26T17:03:33Z
Enable parallel testing for Elephas IO (JENA-1407)
Using parallelism of 4 reduces build time for Elephas IO to
approximately 3-4 minutes in my testing. Also fixed one test that was
not parallel safe to use the pre-existing temporary folder.
----
---
[GitHub] jena issue #297: Enable parallel testing for Elephas IO (JENA-1407)
Posted by kinow <gi...@git.apache.org>.
Github user kinow commented on the issue:
https://github.com/apache/jena/pull/297
Tested on a 4 core, 6GB, Ubuntu LTS VM, with a simple `mvn clean test install`. First I applied the pull request #295 in order to get the build passing.
Then manually applied the diff in this pull request. Before the patch, I got the following for Elephas.
```
[INFO] Apache Jena - Elephas .............................. SUCCESS [ 0.125 s]
[INFO] Apache Jena - Elephas - Common API ................. SUCCESS [ 4.346 s]
[INFO] Apache Jena - Elephas - I/O ........................ SUCCESS [15:46 min]
[INFO] Apache Jena - Elephas - Map/Reduce ................. SUCCESS [ 22.584 s]
[INFO] Apache Jena - Elephas - Statistics Demo App ........ SUCCESS [ 5.769 s]
[INFO] Apache Jena - OSGi ................................. SUCCESS [ 0.098 s]
[INFO] Apache Jena - OSGi bundle .......................... SUCCESS [ 9.213 s]
[INFO] Apache Jena - OSGi Karaf features .................. SUCCESS [ 4.971 s]
[INFO] Apache Jena ........................................ SUCCESS [ 0.503 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 36:11 min
[INFO] Finished at: 2017-10-27T11:21:38+13:00
[INFO] Final Memory: 109M/728M
```
After applying the patch.
```
[INFO] Apache Jena - Elephas .............................. SUCCESS [ 0.226 s]
[INFO] Apache Jena - Elephas - Common API ................. SUCCESS [ 4.269 s]
[INFO] Apache Jena - Elephas - I/O ........................ SUCCESS [03:27 min]
[INFO] Apache Jena - Elephas - Map/Reduce ................. SUCCESS [ 20.512 s]
[INFO] Apache Jena - Elephas - Statistics Demo App ........ SUCCESS [ 3.620 s]
[INFO] Apache Jena - OSGi ................................. SUCCESS [ 0.083 s]
[INFO] Apache Jena - OSGi bundle .......................... SUCCESS [ 8.088 s]
[INFO] Apache Jena - OSGi Karaf features .................. SUCCESS [ 0.170 s]
[INFO] Apache Jena ........................................ SUCCESS [ 0.378 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 13:38 min
[INFO] Finished at: 2017-10-27T12:13:42+13:00
[INFO] Final Memory: 167M/739M
```
Hope that helps,
Bruno
---
[GitHub] jena issue #297: Enable parallel testing for Elephas IO (JENA-1407)
Posted by afs <gi...@git.apache.org>.
Github user afs commented on the issue:
https://github.com/apache/jena/pull/297
Total 36:11 min with 15:46 in Elephas I/O
Total 13:38 min with 03:27 in Elephas I/O
23 min reduction, 12 mins from Elephas I/O
Maybe the saving is even more than reported (clearup? setup?)
---
[GitHub] jena issue #297: Enable parallel testing for Elephas IO (JENA-1407)
Posted by rvesse <gi...@git.apache.org>.
Github user rvesse commented on the issue:
https://github.com/apache/jena/pull/297
Parallelism of 2 gives similar benefits. I have a quad core machine personally.
One possible approach is to use the `build-helper:cpu-count` plugin - http://www.mojohaus.org/build-helper-maven-plugin/cpu-count-mojo.html - to dynamically set the level of parallelism so that we don't over subscribe the users system
---
[GitHub] jena pull request #297: Enable parallel testing for Elephas IO (JENA-1407)
Posted by rvesse <gi...@git.apache.org>.
Github user rvesse closed the pull request at:
https://github.com/apache/jena/pull/297
---
[GitHub] jena issue #297: Enable parallel testing for Elephas IO (JENA-1407)
Posted by rvesse <gi...@git.apache.org>.
Github user rvesse commented on the issue:
https://github.com/apache/jena/pull/297
I have now changed this to dynamically set test parallelism to use 50% of available cores
---
[GitHub] jena issue #297: Enable parallel testing for Elephas IO (JENA-1407)
Posted by afs <gi...@git.apache.org>.
Github user afs commented on the issue:
https://github.com/apache/jena/pull/297
The changes do make the build significantly more usable! Thanks.
---
[GitHub] jena issue #297: Enable parallel testing for Elephas IO (JENA-1407)
Posted by afs <gi...@git.apache.org>.
Github user afs commented on the issue:
https://github.com/apache/jena/pull/297
From the documentation, the default mode of Surefire is that `<threadCount>` is count-per-core.
{noformat}
perCoreThreadCount (Default: true)
(JUnit 4.7 provider) Indicates that threadCount, threadCountSuites,
threadCountClasses, threadCountMethods are per cpu core.
User property: perCoreThreadCount
{noformat}
So setting thread count to 50% CPUs is a number per core?
Let's go with the original PR for the 3.5.0 RC2.
---
[GitHub] jena issue #297: Enable parallel testing for Elephas IO (JENA-1407)
Posted by rvesse <gi...@git.apache.org>.
Github user rvesse commented on the issue:
https://github.com/apache/jena/pull/297
I think leaving at 2 is good, I was playing with using `<perCoreThreadCount>false</perCoreThreadCount>` though obviously that lessens the benefit. I think we can close this out though if we've delivered noticeable improvements already, not really worth tweaking the configs too much as any further gain is likely limited
---
[GitHub] jena issue #297: Enable parallel testing for Elephas IO (JENA-1407)
Posted by afs <gi...@git.apache.org>.
Github user afs commented on the issue:
https://github.com/apache/jena/pull/297
Will try it out ASAP.
How many cores do you have?
If the parallelism is 2, how much effect is there?
I'm wondering if it consumes the whole machine for a few minutes, which might be fine. ATM the user can go do something unrelated (like read email!).
---
[GitHub] jena issue #297: Enable parallel testing for Elephas IO (JENA-1407)
Posted by afs <gi...@git.apache.org>.
Github user afs commented on the issue:
https://github.com/apache/jena/pull/297
Jena 3.5.0 went out with `<threadCount>2</threadCount>` which is per-core (hyperthreading seems to count as 2 for a total of 4 per CPU-chip). `threadCount=1` made a big difference and a setting of 2 more so. Changing 2 to 4 made little difference. [Write-up on the JIRA ticket](https://issues.apache.org/jira/browse/JENA-1407?focusedCommentId=16222769#comment-16222769).
@rvesse With that, are you happy to close the JIRA and this PR? Do you want to bump to 4 anyway?
---