You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@bookkeeper.apache.org by GitBox <gi...@apache.org> on 2020/03/05 10:59:44 UTC
[GitHub] [bookkeeper] Ghatage opened a new pull request #2278: Add forkCount
option to parallelize build and tests
Ghatage opened a new pull request #2278: Add forkCount option to parallelize build and tests
URL: https://github.com/apache/bookkeeper/pull/2278
### Motivation
Current code forks a new JVM per module. (bookkeeper-server, bookkeeper-proto etc)
This means one fork per module for build and testing, no parallelism within the module where majority of the time goes.
We need parallelism within a module during the test execution so we can have the builds complete faster and have the artifacts shipped out quicker.
### Changes
We use the maven surefire plugin but don't define the `forkCount` and hence set it to default of 1.
This means it executes each module with one thread.
This change sets `forkCount` to 5, enabling parallelism in testing and drastically reducing total turnaround time. (by about 2/3rds!)
*Total build+test time without this change*
```
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:06 h
[INFO] Finished at: 2020-03-05T02:01:29-08:00
[INFO] ------------------------------------------------------------------------
```
*Total build+test time with this change*
```
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 18:23 min
[INFO] Finished at: 2020-03-05T02:38:22-08:00
[INFO] ------------------------------------------------------------------------
```
### Things to watch
Added parallelism may cause some flappers but with much trial and error I have come to the number `5`. The flappers are usually only from conflict in obtaining the same port number.
If needed, we can increase the retryCount, but as of now I consistently don't see any flappers at a `forkCount` of 5
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [bookkeeper] Ghatage removed a comment on issue #2278: Add
forkCount option to parallelize build and tests
Posted by GitBox <gi...@apache.org>.
Ghatage removed a comment on issue #2278: Add forkCount option to parallelize build and tests
URL: https://github.com/apache/bookkeeper/pull/2278#issuecomment-602815174
Hi @eolivelli,
Do you think that is necessary? Only asking as [Pulsar doesn't do that either.](https://github.com/apache/pulsar/blob/ca7bb998436a32f7f919567d4b62d42b994363f8/pom.xml#L1238)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [bookkeeper] Ghatage commented on issue #2278: Add forkCount option
to parallelize build and tests
Posted by GitBox <gi...@apache.org>.
Ghatage commented on issue #2278: Add forkCount option to parallelize build and tests
URL: https://github.com/apache/bookkeeper/pull/2278#issuecomment-606930770
Fair enough @eolivelli. I have moved it to 1.
Please note that folks will be able to take benefit of this addition by changing `<forkCount.variable>` locally in their pom.xml
Hopefully his should help the Github Actions Integration tests.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [bookkeeper] Ghatage commented on issue #2278: Add forkCount option
to parallelize build and tests
Posted by GitBox <gi...@apache.org>.
Ghatage commented on issue #2278: Add forkCount option to parallelize build and tests
URL: https://github.com/apache/bookkeeper/pull/2278#issuecomment-604148281
Will do. We can reduce it as you suggest if it causes too much ruckus.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [bookkeeper] eolivelli commented on issue #2278: Add forkCount
option to parallelize build and tests
Posted by GitBox <gi...@apache.org>.
eolivelli commented on issue #2278: Add forkCount option to parallelize build and tests
URL: https://github.com/apache/bookkeeper/pull/2278#issuecomment-606582126
I have restarted bookie tests and integration tests.
Is this patch affecting integration tests ?
I think we cannot run more than one IT at a time, it will use many resources
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [bookkeeper] Ghatage commented on issue #2278: Add forkCount option
to parallelize build and tests
Posted by GitBox <gi...@apache.org>.
Ghatage commented on issue #2278: Add forkCount option to parallelize build and tests
URL: https://github.com/apache/bookkeeper/pull/2278#issuecomment-595181019
@eolivelli Looks like we've halved the total execution time required to run most CI tests on Github Actions! (it's even better locally, on my MBP its reduced by 2/3rds)
Can you please re-run the flapper? (Since this is Github actions only committers can re-execute the tests)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [bookkeeper] eolivelli commented on issue #2278: Add forkCount
option to parallelize build and tests
Posted by GitBox <gi...@apache.org>.
eolivelli commented on issue #2278: Add forkCount option to parallelize build and tests
URL: https://github.com/apache/bookkeeper/pull/2278#issuecomment-608548962
I have restarted the failing Bookie Tests
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [bookkeeper] Ghatage commented on issue #2278: Add forkCount option
to parallelize build and tests
Posted by GitBox <gi...@apache.org>.
Ghatage commented on issue #2278: Add forkCount option to parallelize build and tests
URL: https://github.com/apache/bookkeeper/pull/2278#issuecomment-602815174
Hi @eolivelli,
Do you think that is necessary? Only asking as [Pulsar doesn't do that either.](https://github.com/apache/pulsar/blob/ca7bb998436a32f7f919567d4b62d42b994363f8/pom.xml#L1238)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services