You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@calcite.apache.org by Vladimir Sitnikov <si...@gmail.com> on 2020/07/25 15:30:30 UTC
CALCITE-4140 Configure Gradle Remote Build Cache to speedup builds
Hi,
What do you think if we configure Gradle Remote Build Cache for Calcite?
(see https://guides.gradle.org/using-build-cache )
It could significantly improve PR and regular builds: 12min -> 9min (see
below).
I'm inclined to polish the caching configuration and merge it provided
there are no objections within a week.
https://github.com/apache/calcite/pull/2081
---
It could cache the results of cacheable tasks across builds. For instance,
checkstyle/javadoc/javac results are perfectly cacheable.
Then, if we don't touch linq4j, Gradle could reuse checkstyle/javadoc/javac
results from a known good build.
The cache does account for Java version, classpath, and so on.
I've configured a cache into my own S3 bucket.
The cache would be populated by GitHub Actions branch-based builds (PR
builds would be read-only)
Recent macOS, JDK14 build took 12:24
https://github.com/apache/calcite/runs/909244400?check_suite_focus=true
The one with build cache took 9:36
https://github.com/apache/calcite/runs/909907954?check_suite_focus=true
S3 cache 05:27 saved (05:27 saved on hits, 618ms wasted on misses), reads:
162, hits: 142, elapsed: 11s, processed: 8 MiB
I'm not sure why estimated time saving says 5:27 while the actual duration
difference is 2:48, however, making PR builds faster is nice anyway.
For some reason Javadoc tasks result in cache miss (see build scan
https://scans.gradle.com/s/b5ibehsnmyqxg/timeline?cacheableFilter=cacheable&hideTimeline&outcomeFilter=SUCCESS,FAILED&sort=longest
),
so net saving could be more than listed above.
Test execution is cached yet as there might be tests that depend on
databases that we can't predict and shouldn't cache.
It might be a good idea to make some of the tests cacheable. I believe it
should be OK to cache tests that depend on Calcite only.
Vladimir
Re: CALCITE-4140 Configure Gradle Remote Build Cache to speedup builds
Posted by Vladimir Sitnikov <si...@gmail.com>.
I've fixed javadoc by adding the task to the cache seed job (Javadoc is not
built by default).
Now the cached build (with tests) takes 8:10 (
https://scans.gradle.com/s/qtbs5qpujlspu ), so it reduces build time from
12min to 8min.
The cost here is increased download size: fully cached build downloads
20MiB.
https://scans.gradle.com/s/qtbs5qpujlspu/performance/buildCache?cacheDetails=WyJyZW1vdGUtSGl0Il0#remote-hit
The top consumers are
4.9MiB :core:javadoc
3.7MiB :core:compileJava
Vladimir
сб, 25 июл. 2020 г. в 18:30, Vladimir Sitnikov <sitnikov.vladimir@gmail.com
>:
> Hi,
>
> What do you think if we configure Gradle Remote Build Cache for Calcite?
> (see https://guides.gradle.org/using-build-cache )
>
> It could significantly improve PR and regular builds: 12min -> 9min (see
> below).
>
> I'm inclined to polish the caching configuration and merge it provided
> there are no objections within a week.
> https://github.com/apache/calcite/pull/2081
>
> ---
>
> It could cache the results of cacheable tasks across builds. For instance,
> checkstyle/javadoc/javac results are perfectly cacheable.
> Then, if we don't touch linq4j, Gradle could reuse
> checkstyle/javadoc/javac results from a known good build.
> The cache does account for Java version, classpath, and so on.
>
> I've configured a cache into my own S3 bucket.
> The cache would be populated by GitHub Actions branch-based builds (PR
> builds would be read-only)
>
> Recent macOS, JDK14 build took 12:24
> https://github.com/apache/calcite/runs/909244400?check_suite_focus=true
> The one with build cache took 9:36
> https://github.com/apache/calcite/runs/909907954?check_suite_focus=true
>
> S3 cache 05:27 saved (05:27 saved on hits, 618ms wasted on misses), reads:
> 162, hits: 142, elapsed: 11s, processed: 8 MiB
> I'm not sure why estimated time saving says 5:27 while the actual duration
> difference is 2:48, however, making PR builds faster is nice anyway.
>
> For some reason Javadoc tasks result in cache miss (see build scan
> https://scans.gradle.com/s/b5ibehsnmyqxg/timeline?cacheableFilter=cacheable&hideTimeline&outcomeFilter=SUCCESS,FAILED&sort=longest ),
> so net saving could be more than listed above.
>
> Test execution is cached yet as there might be tests that depend on
> databases that we can't predict and shouldn't cache.
> It might be a good idea to make some of the tests cacheable. I believe it
> should be OK to cache tests that depend on Calcite only.
>
> Vladimir
>
Re: CALCITE-4140 Configure Gradle Remote Build Cache to speedup builds
Posted by Vladimir Sitnikov <si...@gmail.com>.
>There's https://issues.apache.org/jira/browse/INFRA-20516
>on ASF-managed build cache node.
The infra ticket is still in TBD status :-/
In the meantime, I've discovered that https://wasabi.com/ provide S3 object
storage with 0 costs for network egress.
I've created a bucket there and I configured GitHub Actions CI to populate
the build cache.
If you use a similar Java version (8, 11, or 15), then your builds should
be able to reuse CI results.
For instance, if you rebase onto a newer master that includes linq4j
changes, then
your machine does not need to run Checkstyle and compile it.
Hopefully, we could make the build more cache-friendly which would reduce
overall waste, and
it might enable us to have faster CI feedback and run more tests.
Please let me know what is your experience with the cache.
In the future, we could make unit-tests cacheable.
For instance, PartiallyOrderedSetTest and LatticeTest are compute-intensive.
However, the tests are deterministic, so they always produce the same
results provided the test classpath is the same.
Those types of tests look like a good fit for caching.
Just in case, the cache can be disabled with -Ps3.build.cache=false.
Vladimir
Re: CALCITE-4140 Configure Gradle Remote Build Cache to speedup builds
Posted by Vladimir Sitnikov <si...@gmail.com>.
> the cache when merged would still be in your
>personal S3 bucket?
That's true.
There's https://issues.apache.org/jira/browse/INFRA-20516
on ASF-managed build cache node.
Vladimir
Re: CALCITE-4140 Configure Gradle Remote Build Cache to speedup builds
Posted by Michael Mior <mm...@apache.org>.
Seems like a nice performance improvement! Am I correct in
understanding that the cache when merged would still be in your
personal S3 bucket? If that is the case, I wonder if we can instead
connect with INFRA to see about the possibility for ASF-managed
storage.
--
Michael Mior
mmior@apache.org
Le sam. 25 juil. 2020 à 11:31, Vladimir Sitnikov
<si...@gmail.com> a écrit :
>
> Hi,
>
> What do you think if we configure Gradle Remote Build Cache for Calcite?
> (see https://guides.gradle.org/using-build-cache )
>
> It could significantly improve PR and regular builds: 12min -> 9min (see
> below).
>
> I'm inclined to polish the caching configuration and merge it provided
> there are no objections within a week.
> https://github.com/apache/calcite/pull/2081
>
> ---
>
> It could cache the results of cacheable tasks across builds. For instance,
> checkstyle/javadoc/javac results are perfectly cacheable.
> Then, if we don't touch linq4j, Gradle could reuse checkstyle/javadoc/javac
> results from a known good build.
> The cache does account for Java version, classpath, and so on.
>
> I've configured a cache into my own S3 bucket.
> The cache would be populated by GitHub Actions branch-based builds (PR
> builds would be read-only)
>
> Recent macOS, JDK14 build took 12:24
> https://github.com/apache/calcite/runs/909244400?check_suite_focus=true
> The one with build cache took 9:36
> https://github.com/apache/calcite/runs/909907954?check_suite_focus=true
>
> S3 cache 05:27 saved (05:27 saved on hits, 618ms wasted on misses), reads:
> 162, hits: 142, elapsed: 11s, processed: 8 MiB
> I'm not sure why estimated time saving says 5:27 while the actual duration
> difference is 2:48, however, making PR builds faster is nice anyway.
>
> For some reason Javadoc tasks result in cache miss (see build scan
> https://scans.gradle.com/s/b5ibehsnmyqxg/timeline?cacheableFilter=cacheable&hideTimeline&outcomeFilter=SUCCESS,FAILED&sort=longest
> ),
> so net saving could be more than listed above.
>
> Test execution is cached yet as there might be tests that depend on
> databases that we can't predict and shouldn't cache.
> It might be a good idea to make some of the tests cacheable. I believe it
> should be OK to cache tests that depend on Calcite only.
>
> Vladimir