You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@samza.apache.org by shanthoosh <gi...@git.apache.org> on 2018/09/15 04:19:40 UTC
[GitHub] samza pull request #640: [WIP][Not ready for review]SAMZA-1885: Improve samz...
GitHub user shanthoosh opened a pull request:
https://github.com/apache/samza/pull/640
[WIP][Not ready for review]SAMZA-1885: Improve samza build time.
Samza codebase currently uses gradle 2.8 (which was released in 2014) and builds the codebase serially executing each of the gradle tasks one at a time. Not taking advantage of recent gradle features has caused slow builds (taking around 15 minutes to build samza codebase locally). This hampers the overall developer productivity within the team. Proposal is to have a fasten builds by a series of changes.
**Changes:**
1. Upgrade to gradle version: 4.7(changes in gradle.properties, gradle-wrapper.jar).
2. Enable parallel gradle builds.
3. Set maximumParallelForks to 15(The maximum number of forked test processes to execute in parallel. The default value is 1 (no parallel test execution). It cannot exceed the value of max-workers for the current build.
Reference: https://docs.gradle.org/current/dsl/org.gradle.api.tasks.testing.Test.html)
4. Fix the tests that cannot be run in parallel(TestsJobsResource, TestTaskResource).
**Result:**
Reduces the build time by a factor of 2.
Before these changes, it took around 13 minutes 40 seconds to build the open source samza codebase locally.
After these changes, it took around around 6 minutes 14 seconds to build samza codebase locally.
**Pending:**
Ran the build locally multiple(50) times. After these changes, only the TestRunner API tests fail consistently due to static state in it to maintain inemory(Will fix it in subsequent patches).
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/shanthoosh/samza improve_build_time
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/samza/pull/640.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #640
----
commit 347f8fef986ffaf87b6275d72370aea0f6ac9daf
Author: Shanthoosh Venkataraman <sp...@...>
Date: 2018-09-14T17:43:20Z
Changes:
1. Switch to newer version of gradle.
2. Turn on gradle parallel builds.
----
---
[GitHub] samza pull request #640: [WIP][Not ready for review]SAMZA-1885: Improve samz...
Posted by shanthoosh <gi...@git.apache.org>.
Github user shanthoosh closed the pull request at:
https://github.com/apache/samza/pull/640
---
[GitHub] samza pull request #640: [WIP][Not ready for review]SAMZA-1885: Improve samz...
Posted by shanthoosh <gi...@git.apache.org>.
Github user shanthoosh closed the pull request at:
https://github.com/apache/samza/pull/640
---
[GitHub] samza pull request #640: [WIP][Not ready for review]SAMZA-1885: Improve samz...
Posted by shanthoosh <gi...@git.apache.org>.
GitHub user shanthoosh reopened a pull request:
https://github.com/apache/samza/pull/640
[WIP][Not ready for review]SAMZA-1885: Improve samza build time.
Samza codebase currently uses gradle 2.8 (which was released in 2014) and builds the codebase serially executing each of the gradle tasks one at a time. Not taking advantage of recent gradle features has caused slow builds (taking around 15 minutes to build samza codebase locally). This hampers the overall developer productivity within the team.
**Changes:**
1. Upgrade to gradle version: 4.7(changes in gradle.properties, gradle-wrapper.jar).
2. Enable parallel gradle builds.
3. Set maximumParallelForks to 5(The maximum number of forked test processes to execute in parallel. The default value is 1 (no parallel test execution). It cannot exceed the value of max-workers for the current build.
Reference: https://docs.gradle.org/current/dsl/org.gradle.api.tasks.testing.Test.html)
4. Fix the tests that cannot be run in parallel(TestsJobsResource, TestTaskResource).
**Result:**
Reduces the build time by a factor of 2.
Before these changes, it took around 13 minutes 40 seconds to build the open source samza codebase locally.
After these changes, it took around around 6 minutes 14 seconds to build samza codebase locally.
**Pending:**
Ran the build locally multiple(50) times. After these changes, only the TestRunner API tests fail consistently due to static partition state held in it(Will fix it as a part of this patch).
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/shanthoosh/samza improve_build_time
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/samza/pull/640.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #640
----
commit b48a116bb7a91beeac014ae30f963555332223ef
Author: Shanthoosh Venkataraman <sv...@...>
Date: 2018-09-15T18:43:08Z
Changes:
1. Switch to newer version of gradle.
2. Turn on gradle parallel builds.
----
---