You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@maven.apache.org by ol...@apache.org on 2023/05/31 11:10:05 UTC
[maven-build-cache-extension] branch master updated: Cache documentation improvements (#40)
This is an automated email from the ASF dual-hosted git repository.
olamy pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/maven-build-cache-extension.git
The following commit(s) were added to refs/heads/master by this push:
new bcc11a0 Cache documentation improvements (#40)
bcc11a0 is described below
commit bcc11a04a7dc822009a6182cf52220ea76cc2977
Author: Alex Ashitkin <as...@gmail.com>
AuthorDate: Wed May 31 07:09:59 2023 -0400
Cache documentation improvements (#40)
Co-authored-by: aashitki <al...@db.com>
---
src/site/markdown/getting-started.md.vm | 51 ++++----
src/site/markdown/how-to.md | 46 ++++---
src/site/markdown/index.md | 138 +++++++++++----------
src/site/markdown/parameters.md | 28 +++--
src/site/markdown/performance.md | 41 +++----
src/site/markdown/remote-cache.md | 209 +++++++++++++++++---------------
src/site/markdown/usage.md | 49 ++++----
7 files changed, 294 insertions(+), 268 deletions(-)
diff --git a/src/site/markdown/getting-started.md.vm b/src/site/markdown/getting-started.md.vm
index 4f9e230..07ba2a5 100644
--- a/src/site/markdown/getting-started.md.vm
+++ b/src/site/markdown/getting-started.md.vm
@@ -18,16 +18,17 @@
Getting Started
---------------
-To on-board incremental Maven you need to complete several steps:
+To onboard the build-cache extension you need to complete several steps:
* Declare caching extension in your project (either in `pom.xml` or `.mvn/extensions.xml`)
-* Add `maven-build-cache-config.xml` cache config in `.mvn/` (optional) to customize default behavior
-* Validate build results and iteratively, adjust config to properly reflect project specifics
+* Add `maven-build-cache-config.xml` cache config in `.mvn/` (optional) to customize the default behavior
+* Validate build results and iteratively adjust config to reflect project specifics properly
* Setup remote cache (optional)
#[[###]]# Declaring build cache extension
```xml
+
<extension>
<groupId>org.apache.maven.extensions</groupId>
<artifactId>maven-build-cache-extension</artifactId>
@@ -35,41 +36,49 @@ To on-board incremental Maven you need to complete several steps:
</extension>
```
-either in `pom.xml`'s `<project>/<build>/<extensions>` or in `.mvn/extensions.xml`'s `<extensions>`
+either in `pom.xml`'s `<project>/<build>/<extensions>` or in `.mvn/extensions.xml`'s `<extensions>`. Using core
+extension model (`.mvn/extensions.xml` file) is preferable as it allows better access to maven APIs and could allow
+more sophisticated optimizations in the future.
#[[###]]# Adding build cache config
Copy template config [`maven-build-cache-config.xml`](../resources/maven-build-cache-config.xml)
to [`.mvn/`](https://maven.apache.org/configure.html) directory of your project.
-To get overall understanding of build cache machinery, it is recommended to review the config and read comments. In typical
-scenario you need to:
+To understand the caching machinery, review the config and read the comments. In a typical scenario, you need to:
-* Exclude unstable, temporary files or environment specific files
-* Add plugins reconciliation rules – add critical plugins parameters to reconciliation
-* Configure precise source code files selectors. Though source code locations discovered automatically from project and plugins config,
- there might be edge cases.
-* Add remote cache location (if remote cache is used)
+* Exclude unstable, temporary files or environment-specific files
+* Add critical plugins parameters to runtime reconciliation
+* Configure precise source code file selectors. Though source code locations are discovered automatically from project
+ and
+ plugin configs, there might be edge cases.
+* Configure remote cache (if using the remote cache)
#[[###]]# Adjusting build cache config
-Having extension run usual command, like `mvn package`. Verify the caching engine is activated:
+After configuring the extension, run a usual command, for example, `mvn package`, and verify the caching engine is
+activated:
-* Check log output - there should be cache related output or initialization error message:
+* Check log output - there should be the cache-related output or initialization error message:
```
[INFO] Loading cache configuration from <project dir>/.mvn/maven-build-cache-config.xml
```
* Navigate to your local repo directory - there should be a sibling directory `build-cache` next to the usual
local `repository`.
-* Find `buildinfo.xml` in the cache repository for typical module and review it. Ensure that
- * expected source code files are present in the build info
- * Review all plugings used in the build and add their critical parameters to reconciliation
+* Find `buildinfo.xml` in the cache repository for a typical module and review it. Ensure that
+ * Expected source code files are present in the build info
+ * Review all plugins used in the build and add their critical parameters to the reconciliation
-It is recommended to find the best working trade-off between fairness and cache efficiency. Adding unnecessary rules and
+Try to find the best working trade-off between fairness and cache efficiency. Adding unnecessary rules and
checks could reduce both performance and cache efficiency (hit rate).
#[[###]]# Adding caching CI and remote cache
-To leverage remote cache feature there should a shared storage provide. Any technology supported
-by [Maven Resolver](https://maven.apache.org/resolver/) will suffice. In simplest form it could be a http web server which
-supports get/put operations ([Nginx OSS](http://nginx.org/en/) with fs module or any other equivalent).
-See [Remote cache setup](remote-cache.html) for detailed description of cache setup.
+To leverage the remote cache feature need to provide shared storage. Any technology supported
+by [Maven Resolver](https://maven.apache.org/resolver/) will suffice. In the simplest form, it could be any HTTP web
+server
+supporting get/put operations. Working examples:
+
+* Artifactory (generic repository)
+* [Nginx OSS](http://nginx.org/en/) with fs module
+
+See [Remote cache setup](remote-cache.html) for a detailed description of the remote cache setup
diff --git a/src/site/markdown/how-to.md b/src/site/markdown/how-to.md
index 08c2edb..4c1a84f 100644
--- a/src/site/markdown/how-to.md
+++ b/src/site/markdown/how-to.md
@@ -17,12 +17,12 @@
## Overview
-Cache configuration provides you additional control over incremental Maven behavior. Follow it step by step to
-understand how it works and figure out your optimal config
+Cache configuration provides you additional control over the build-cache extension behavior. Follow it step-by-step to
+understand how it works, and figure out an optimal config
### Minimal config
-Absolutely minimal config
+Minimal config
```xml
<?xml version="1.0" encoding="UTF-8" ?>
@@ -58,7 +58,7 @@ Just add `<remote>` section under `<configuration>`
### Adding more file types to input
-Add all the project specific source code files in `<glob>`. Scala in this case:
+Add all the project-specific source code files in `<glob>`. Scala, in this case:
```xml
<input>
@@ -70,8 +70,7 @@ Add all the project specific source code files in `<glob>`. Scala in this case:
### Adding source directory for bespoke project layouts
-In most of the cases incremental Maven will recognize directories automatically by build introspection. If not, you can
-add additional directories with `<include>`. Also you can filter out undesirable dirs and files by using exclude tag
+In most cases, the build-cache extension automatically recognizes directories by introspecting the build. When it is not enough, adding additional directories with `<include>` is possible. Also, you can filter out undesirable dirs and files by using exclude tag.
```xml
<input>
@@ -87,7 +86,7 @@ add additional directories with `<include>`. Also you can filter out undesirable
</input>
```
-### Plugin property is env specific and yields different cache key in different environments
+### Plugin property is environment-specific and yields different cache keys in different environments
Consider to exclude env specific properties:
@@ -108,11 +107,11 @@ Consider to exclude env specific properties:
</input>
```
-Implications - builds with different `argLine` will have identical key. Validate that is semantically valid.
+Implications - builds with different `argLine` will have an identical key. Validate that it is acceptable in terms of artifact equivalency.
-### Plugin property points to directory where only subset of files is relevant
+### Plugin property points to a directory where only a subset of files is relevant
-If plugin configuration property points to `somedir` it will be scanned with default glob. You can tweak it with custom
+If the plugin configuration property points to `somedir`, it will be scanned with the default glob. You can tweak it with custom
processing rule
```xml
@@ -133,9 +132,9 @@ processing rule
</input>
```
-### Local repository is not updated because `install` is cached
+### Local repository is not updated because the `install` phase is cached
-Add `executionControl/runAlways` section
+Add `executionControl/runAlways` section:
```xml
<executionControl>
@@ -161,10 +160,10 @@ Add `executionControl/runAlways` section
</executionControl>
```
-### I occasionally cached build with `-DskipTests=true` and tests do not run now
+### I occasionally cached build with `-DskipTests=true`, and tests do not run now
-If you add command line flags to your build, they do not participate in effective pom - Maven defers final value
-resolution to plugin runtime. To invalidate build if filed value is different in runtime, add reconciliation section
+If you add command line flags to your build, they do not participate in effective pom - Maven defers the final value
+resolution to plugin runtime. To invalidate the build if the filed value is different in runtime, add a reconciliation section
to `executionControl`:
```xml
@@ -194,27 +193,24 @@ to `executionControl`:
</cache>
```
-Please notice `skipValue` attribute. It denotes value which forces skipped execution.
-Read `propertyName="skipTests" skipValue="true"` as if property skipTests has value true, plugin will skip execution If
-you declare such value incremental Maven will reuse appropriate full-build though technically they are different, but
-because full-build is better it is safe to reuse
+Please notice the `skipValue` attribute. It captures the value that makes the plugin skip execution. Think of `skipProperty` as follows: if the build started with `-DskipTests=true`, restoring results from a build with completed tests is safe because the local run does not require completed tests. The same logic applies to any other plugin, not just Surefire.
### How to renormalize line endings in working copy after committing .gitattributes (git 2.16+)
-Ensure you've committed (and ideally pushed everything) - no changes in working copy. After that:
+Ensure you've committed (and ideally pushed everything) - no changes in the working copy. After that:
```shell
-# Rewrite objects and update index
+# Rewrite objects and update the index
git add --renormalize .
# Commit changes
-git commit -m "Normalizing line endings"
-# Remove working copy paths from git cache
+git commit -m "Normalizing line endings."
+# Remove working copy paths from the git cache
git rm --cached -r .
# Refresh with new line endings
git reset --hard
```
-### I want to cache interim build and override it later with final version
+### I want to cache the interim build and override it later with the final version
-Solution: set `-Dmaven.build.cache.remote.save.final=true` to nodes which produce final builds. Such builds will not be overridden
+Solution: set `-Dmaven.build.cache.remote.save.final=true` to nodes that produce final builds. Such builds will not be overridden
and eventually will replace all interim builds
diff --git a/src/site/markdown/index.md b/src/site/markdown/index.md
index 6c15811..6f7464d 100644
--- a/src/site/markdown/index.md
+++ b/src/site/markdown/index.md
@@ -19,102 +19,111 @@
Build cache is an extension targeted to simplify and make more efficient work with large builds in Maven.
-That is achieved by a combination of features:
+A combination of features achieves that:
-* Incremental builds over the changed project graph part only
-* Subtree support in multimodule projects (caches discovered from the larger project)
+* Incremental builds works on the modified part of the project graph part only
+* Subtree support for multimodule projects to work on the part of the codebase in isolation
* Version normalization to support project version agnostic caches
* Project state restoration (partial) to avoid expensive tasks (code generation and similar)
-Large projects usually pose scalability challenges and work with such projects require build tool which scales. Cache
-extension addresses that with incremental build execution and ability to efficiently work on sub-parts of a larger
+Large projects usually pose scalability challenges, and working with such projects requires a build tool that scales.
+The cache
+extension addresses that with incremental build execution and the ability to efficiently work on sub-parts of a larger
project without building and installing dependencies from the larger project. Though, features implemented in Maven
-should give noticeable benefits in medium and small sized projects as well.
+should also give noticeable benefits in medium and small-sized projects.
### Cache concepts
-The idea of the build cache is to calculate key from module inputs, store outputs in cache and restore them later
-transparently to the standard Maven core. In order to calculate the key cache engine analyzes source code, build flow,
-plugins and their parameters. This allows to deterministically associate each project state with unique key and restore
-up-to-date (not changed) projects from cache and rebuild out-of-date(changed) ones. Restoring artifacts associated with
-a particular project state improves build times by avoiding re-building unnecessary modules. Cache does not make any
-interventions to actual build execution process and fully delegates build work to Maven core. This ensures that
-artifacts produced in presence of cache are equivalent to result produced by a standard Maven build.
-To achieve accurate key calculation incremental Maven combines automatic introspection
-of [project object model](https://maven.apache.org/pom.html#What_is_the_POM) and allows fine-grained tuning by means of
-configuration file and xml attributes. Source code content fingerprinting is digests based which is more reliable over
-widely used file timestamps in tools like Make or Apache Ant. Deterministic build state allows reliably cache outputs
-even of the build in progress and share them between teams using remote cache. Deterministic inputs calculation allows
-distributed and parallel builds running in heterogeneous environments (like cloud of build agents)
-could efficiently reuse cached build artifacts. Therefore, incremental Maven is particularly well-suited for large Maven
-projects that have significant number of small modules. Remote cache in conjunction with relocatable inputs
-identification effectively enables "change once - build once" approach across all environments.
+The idea of the build cache is to calculate a key from module inputs, store outputs in the cache, and restore them later
+transparently to the standard Maven core. The cache deterministically associates each project state with a unique key
+and restores it in subsequent builds. It analyzes source code, project model,
+plugins, and their parameters. Projects with the same key are up-to-date (not changed) and could be safely restored from
+the cache. Projects producing different keys are out-of-date (changed), and the cache fully rebuilds them. In the latter
+case, the cache does not make any
+interventions to the build execution logic and delegates build work to the standard maven Maven core. This approach
+ensures that
+artifacts produced in the presence of a cache are equivalent to the result produced by a standard Maven build.
+To achieve an accurate key calculation, the build-cache extension combines automatic introspection
+of [project object model](https://maven.apache.org/pom.html#What_is_the_POM) and fine-grained tuning using
+a configuration file. Source code content fingerprinting is digests based, which is more reliable over
+widely used file timestamps in tools like Make or Apache Ant. Cache outputs could be shared using a remote cache.
+Deterministic inputs calculation allows distributed and parallel builds running in heterogeneous environments (like a
+cloud of build agents) efficiently reuse cached build artifacts as soon as they are published. Therefore, incremental
+Maven is particularly well-suited for large Maven
+projects that have a significant number of small modules. Remote cache, combined with relocatable inputs
+identification, effectively enables the "change once - build once" approach across all environments.
### Maven insights
-The challenge of implementing build cache in Maven is that domain model is overly generic and doesn't have dedicated api
-for build inputs. Because of that, even 2 identically looking builds from the same source code could normally produce 2
-different results. The question here is tolerance level - can you accept particular discrepancies or not. For most of
-teams artifacts produced in the same build environment from the same source code will be considered equivalent and
-technical differences between them (like different timestamps in jar manifests) could be ignored. Now consider scenario
-when artifact is first produced with compiler X and cached. Later, without touching source code, compiler changes to Y
-and build yields significantly different outcomes of compilation. Should the produced artifacts be considered as
-equivalent? Both Yes and No answers are possible and could be even desirable in different scenarios. When productivity
-and performance are the primary concerns it could be desirable to tolerate insignificant discrepancies and maximise
-reuse of cached builds. As long as correctness is in focus there could be demand to comply with the exact release
-process. In the same way as with classic Maven, correctness is ensured by proper build configuration and controllable
-build environments. In the same way as with classic Maven the previous build is just an approximation of today build
-with some tolerance (implementation, configuration and environment driven).
+Maven is a proven tool with a long history and core design established many years ago. Historically, Maven's core was
+designed with generic stable interfaces that don't have a concept of inputs and outputs. It just runs as configured, but
+the core does not control the inputs and effects of the run. Most commonly, artifacts produced in the same build
+environment from the same source code will be considered equivalent. But even two identically looking builds from the
+same source code could have two different results. The question here is tolerance level - can you accept particular
+discrepancies? Though technical differences between artifacts like timestamps in manifests are largely ignored, when
+compilers used are of different levels, it is likely a critical difference. Should the produced artifacts be considered
+equivalents? Yes and No answers are possible and could be desirable in different scenarios. When productivity
+and performance are the primary concerns, it could be beneficial to tolerate insignificant discrepancies and maximize
+the reuse. As long as correctness is in focus, there could be a demand to comply with the exact release requirements. In
+the same way as Maven, the cache correctness is ensured by proper build configuration and control over the build
+environment. As Maven itself, the cached result is just an approximation of another build with some tolerance level
+(implementation, configuration, and environment driven) that must be configured to meet your needs.
### Implementation insights
-At very simple form, the build cache Maven is essentially a hash function which takes Maven project and produces cache
-key for a project. Then the key is used to store and restore build results. Because of different factors there could be
-collisions and instabilities in the produced key. Collision could happen if the same key produced from the semantically
-different build states and will result in unintended reuse. Instability means that same input yields different key in
-different runs resulting in cache misses. The ultimate target is to find tradeoff between correctness and performance by
-means of configuration. In current implementation this is achieved by configuring cache processing rules in xml file.
+Simply put, the build cache is a hash function that takes a Maven project and produces a unique key. Then the key is
+used to store and restore build results. Because of different factors, there could be
+collisions and instabilities in the produced key. A collision happens when the semantically different builds have the
+same key and will result in unintended reuse. Instability means that the same input yields different keys resulting in
+cache misses. The ultimate target is to find a tradeoff between correctness and performance by configuring cache
+processing rules in an xml file.
-In order to achieve better correctness need to:
+To maximize correctness:
-* Verify that every relevant file is selected as input to engine
-* Add critical plugin parameters to reconciliation (because they could be overridden from command line)
+* Select every relevant file as input to the engine
+* Add all the functional plugin parameters to the reconciliation
-In order to achieve better reuse need to:
+To maximize reuse you need to:
-* ensure that non-critical files (test logs, readme and similar) are filtered out from build inputs.
-* non-critical plugin parameters (like number of threads in build) are filtered out from build inputs
-* Source code is relocatable and build parameters are relocatable (not environment specific)
+* Filter out non-essential files (documentation, IDE configs, and similar)
+* Minimize the overall number of controlled plugin parameters and exclude behavioral plugin parameters (like the number
+ of threads or log level)
+* Make source code relocatable (environment agnostic)
-Essentially cache setup is a process of inspecting build, taking these decision and reflect them in the cache
+Effectively, cache setup involves inspecting the build, taking these decisions, and reflecting them in the cache
configuration.
-Please notice though idea of perfectly matching builds might be tempting, but it is not practical with regard to
-caching. Perfect correctness could lead to prevailing hit misses and render caching useless when applied to real
-projects. In practice, configuring sufficient(good enough) correctness might yield the best outcomes. Incremental Maven
-provides flexible and transparent control over caching policy and allows achieving desired outcomes - maximize usability
-or maximize equivalence(correctness) between pre-cached candidates and requested builds.
+Though strict, comprehensive cache rules aiming for 100% coverage of all parameters and files might be tempting, it is
+rarely the optimal decision. When applied to real projects, perfect correctness could lead to prevailing hit misses and
+render caching useless. Configuring sufficient (good enough) correctness might yield the best outcomes. Incremental
+Maven
+provides flexible and transparent control over caching policy and allows achieving desired results - maximizes usability
+or maximize equivalency between pre-cached candidates and requested builds.
## Usage
-Cache extension is an opt-in feature. It is delivered as is and though the tool went through careful verification it's
-still build owner's responsibility to verify build outcomes.
+Cache extension is an opt-in feature. It is delivered as is, and though the tool went through careful verification, it's
+still the build owner's responsibility to verify build outcomes.
### Recommended Scenarios
-Given all the information above, the Incremental Maven is recommended to use in scenarios when productivity and
+Given all the information above, the build-cache extension is recommended to use in scenarios when productivity and
performance are in priority. Typical cases are:
-* Continuous integration. In conjunction with remote cache, the extension could drastically reduce build times,
- validate pull requests faster and reduce load on CI nodes
-* Speedup developer builds. By reusing cached builds developers could verify changes much faster and be more productive.
+* Continuous integration. In conjunction with the remote cache, the extension could drastically reduce build times,
+ validate pull requests faster and reduce the load on CI nodes
+* Speedup developer builds. By reusing cached builds, developers could verify changes much faster and be more
+ productive.
No more `-DskipTests` and similar.
-* Assemble artifacts faster. In some development models it might be critical to have as fast build/deploy cycle as
- possible. Caching helps to cut down time drastically in such scenarios because it doesn't require to build cached
+* Assemble artifacts faster. In some development models, it might be critical to have a build/deploy turnaround as fast
+ as
+ possible. Caching helps to cut down time drastically in such scenarios because it doesn't require building cached
dependencies.
-For cases there correctness must be ensured (eg prod builds), it is recommended to disable cache and do clean builds.
-Such scheme allows to validate cache correctness by reconciling outcomes of cached builds against the reference builds.
+For cases when users must ensure the correctness (e.g. prod builds), it is recommended to disable the cache and do clean
+builds instead.
+Such a scheme allows cache correctness validation by reconciling the outcomes of cached builds against the reference
+builds.
## See also
@@ -125,4 +134,3 @@ Such scheme allows to validate cache correctness by reconciling outcomes of cach
* [Performance](performance.html) - performance tuning
* [Cache Parameters](parameters.html) - description of supported parameters
* [Sample config file](maven-build-cache-config.xml)
-
diff --git a/src/site/markdown/parameters.md b/src/site/markdown/parameters.md
index d3c8f89..c10ef27 100644
--- a/src/site/markdown/parameters.md
+++ b/src/site/markdown/parameters.md
@@ -17,31 +17,33 @@
## Build Cache Parameters
-This documents contains various configuration parameters supported by cache engine
+This document contains various configuration parameters supported by the cache engine.
### Command line flags
| Parameter | Description | Usage Scenario |
|------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------|
| `-Dmaven.build.cache.configPath=path to file` | Location of cache configuration file | Cache config is not in default location |
-| `-Dmaven.build.cache.enabled=(true/false)` | Cache and associated features disabled/enabled | To remove noise from logs then remote cache is not available |
+| `-Dmaven.build.cache.enabled=(true/false)` | Cache and associated features disabled/enabled | To remove noise from logs when the remote cache is not available |
| `-Dmaven.build.cache.remote.enabled=(true/false)` | Checks and downloads artifacts from the remote cache (overrides <remote enabled=("true"/"false")>) | To control remote cache access by node, if, say, some nodes lack reliable access |
| `-Dmaven.build.cache.remote.save.enabled=(true/false)` | Remote cache save allowed or not | To designate nodes which allowed to push in remote shared cache |
-| `-Dmaven.build.cache.remote.save.final=(true/false)` | Prohibit to override remote cache | To ensure that reference build is not overridden by interim build |
+| `-Dmaven.build.cache.remote.save.final=(true/false)` | Prohibit to override remote cache | Prevents cache records from being overridden by subsequent builds |
| `-Dmaven.build.cache.remote.url=` | Url of the remote cache (overrides <remote><url></url></remote>) | To override url of remote cache from command line |
| `-Dmaven.build.cache.remote.server.id=` | Id of the remote cache server (overrides <remote id=""></remote>) | To override id of remote cache server from command line |
| `-Dmaven.build.cache.failFast=(true/false)` | Fail on the first module which cannot be restored from cache | Remote cache setup/tuning/troubleshooting |
| `-Dmaven.build.cache.baselineUrl=<http url>` | Location of baseline build for comparison | Remote cache setup/tuning/troubleshooting |
| `-Dmaven.build.cache.lazyRestore=(true/false)` | Restore artifacts from remote cache lazily | Performance optimization |
| `-Dmaven.build.cache.restoreGeneratedSources=(true/false)` | Do not restore generated sources and directly attached files | Performance optimization |
-| `-Dmaven.build.cache.alwaysRunPlugins=<list of plugins>` | Comma seprated list of plugins to always run regardless of cache state. An example: `maven-deploy-plugin:*,maven-install-plugin:install` | Remote cache setup/tuning/troubleshooting |
-| `-Dmaven.build.cache.skipCache=(true/false)` | Skip looking up artifacts in caches. Does not affect writing artifacts to caches, disables only reading when set to `true` | May be used to trigger a forced rebuild when matching artifatcs do exist in caches |
+| `-Dmaven.build.cache.alwaysRunPlugins=<list of plugins>` | Comma separated list of plugins to always run regardless of cache state. An example: `maven-deploy-plugin:*,maven-install-plugin:install` | Remote cache setup/tuning/troubleshooting |
+| `-Dmaven.build.cache.skipCache=(true/false)` | Skip looking up artifacts in caches. Does not affect writing artifacts to caches, disables only reading when set to `true` | May be used to trigger a forced rebuild when matching artifacts do exist in caches |
-### Project level properties
+### Project-level properties
-Project level parameters allow overriding global parameters on project level Must be specified as project properties:
+Project-level parameters allowing to override global parameters on the project level. Must be specified as prefixed
+project properties:
```xml
+
<pom>
...
<properties>
@@ -52,9 +54,9 @@ Project level parameters allow overriding global parameters on project level Mus
| Parameter | Description |
|---------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| `maven.build.cache.input.glob` | Project specific glob to select sources. Overrides global glob. |
-| `maven.build.cache.input` | Property prefix to mark paths which must be additionally scanned for source code. Value of property starting with this prefix will be treated as path relatively to current project/module |
-| `maven.build.cache.exclude` | Property prefix to mark paths which must be excluded from source code search. Value of property starting with this prefix will be treated as path to current project/module |
-| `maven.build.cache.processPlugins` | Introspect plugins to find inputs or not. Default is true. |
-| `maven.build.cache.skipCache` | Skip looking up artifacts for a particular project in caches. Default is false. |
-| `maven.build.cache.restoreGeneratedSources` | Restore generated sources and directly attached files. Default is true. |
+| `maven.build.cache.input.glob` | Project specific glob to select sources. Overrides the global glob. |
+| `maven.build.cache.input` | Additional source code locations. Relative paths calculated from the current project/module root |
+| `maven.build.cache.exclude` | Paths to exclude from source code search. Relative paths calculated from the current project/module root |
+| `maven.build.cache.processPlugins` | Introspect plugins to find inputs or not. The default value is true. |
+| `maven.build.cache.skipCache` | Skip looking up artifacts for a particular project in caches. The default value is false. |
+| `maven.build.cache.restoreGeneratedSources` | Restore generated sources and directly attached files. The default value is true. |
diff --git a/src/site/markdown/performance.md b/src/site/markdown/performance.md
index e8e3523..060911a 100644
--- a/src/site/markdown/performance.md
+++ b/src/site/markdown/performance.md
@@ -17,21 +17,15 @@
## Performance Tuning
-Various setup options which affect cache performance.
-
### General notes
-Tuning of cache performance could reduce both resources consumption and build execution time but that is not guaranteed.
-In many scenarios build time of invalidated (changed) projects could be dominating in overall build time. Performance
-wins achieved by a faster cache engine might not correlate with final build times in straightforward way. As usual with
-performance, effect of performance optimizations should be carefully measured in relevant scenarios.
+Cache tuning could significantly reduce resource consumption and build execution time, but that is not guaranteed. Performance-tuning hints might not considerably affect the build time depending on a usage pattern. As usual with performance tuning, measure results in relevant scenarios to validate results and weigh the pros and cons.
### Hash algorithm selection
-By default, cache uses SHA-256 algorithm which is sufficiently fast and provides negligible probability of hash
-collisions. In projects with large codebase, performance of hash algorithms becomes more important and in such
-scenarios [XX](https://cyan4973.github.io/xxHash/) or XXMM (memory mapped files) hashing algorithms provide better
-performance.
+By default, the cache uses the SHA-256 algorithm, which is sufficiently fast and provides a negligible probability of hash
+collisions. In projects with a large codebase, the performance of hash algorithms becomes more critical, and other algorithms like
+[XX](https://cyan4973.github.io/xxHash/) or XXMM (memory-mapped files) could provide better performance.
```xml
<hashAlgorithm>XX</hashAlgorithm>
@@ -43,11 +37,10 @@ or
<hashAlgorithm>XXMM</hashAlgorithm>
```
-### Filter out unnecessary/huge artifacts
+### Filter out unnecessary artifacts
-Price of uploading and downloading from cache of huge artifacts could be significant. In many scenarios assembling WAR,
-EAR or ZIP archive could be done more efficiently locally from cached JARs than storing bundles. In order to filter out
-artifacts add configuration section:
+The price of uploading and downloading huge artifacts could be significant. In many scenarios assembling WAR,
+EAR or ZIP archive locally is more efficient than writing to soring in cache bundles. To filter out artifacts, add the configuration section:
```xml
<cache>
@@ -59,24 +52,21 @@ artifacts add configuration section:
</cache>
```
-### Use lazy restore
+### Use a lazy restore
-By default, cache tries to restore all artifacts for a project preemptively. Lazy restore could give a significant time
-and resources wins for remote cache by avoiding requesting and downloading unnecessary artifacts from cache. Use command
-line flag:
+By default, the cache tries to restore all artifacts for a project preemptively. Lazy restore could give significant time by avoiding requesting and downloading unnecessary artifacts from the cache.
+It is beneficial when small changes are a dominating build pattern. Use command line flag:
```
-Dmaven.build.cache.lazyRestore=true";
```
-Note: In case of cache corruption lazy cache cannot fallback to normal execution, it will fail instead. To heal the
-corrupted cache need to force rewrite of the cache or remove corrupted cache entries manually
+In cache corruption situations, the lazy cache cannot support fallback to normal execution. It will fail instead. To heal the corrupted cache, manually remove corrupted cache entries or force cache rewrite.
### Disable project files restoration
-By default, cache support partial restore of source code state from cached generated sources (and potentially more,
-depending on configuration). This could be helpful in local environment, but likely unnecessary and adds overhead in
-continuous integration. To disable add command line flag
+By default, cache supports the partial restoration of source code state from cached generated sources (and potentially more,
+depending on configuration). It is helpful in a local environment but likely unnecessary and adds overhead in continuous integration. To disable, add a command line flag.
```
-Dmaven.build.cache.restoreGeneratedSources=false";
@@ -84,10 +74,7 @@ continuous integration. To disable add command line flag
### Disable post-processing of archives(JARs, WARs, etc) META-INF
-Post-processing is disabled by default, but for some projects cache could be configured to auto-correct metadata (most
-notably [MANIFEST.MF `Implementation-Version`](https://docs.oracle.com/javase/8/docs/technotes/guides/jar/jar.html#Main_Attributes))
-. This could be rather expensive as it requires copying and repacking archive entries. If metadata state is not relevant
-for the build (continuous integration, `verify` scenarios and similar) consider disabling it:
+The cache could be configured to auto-correct metadata (most notably [MANIFEST.MF `Implementation-Version`](https://docs.oracle.com/javase/8/docs/technotes/guides/jar/jar.html#Main_Attributes)). The correction requires copying and repacking archive entries and adds overhead. If the metadata state is not relevant for the build, consider disabling it (off by default):
```xml
<cache>
diff --git a/src/site/markdown/remote-cache.md b/src/site/markdown/remote-cache.md
index ba0cc36..e31f014 100644
--- a/src/site/markdown/remote-cache.md
+++ b/src/site/markdown/remote-cache.md
@@ -17,88 +17,100 @@
## Overview
-This document describes generic approach to remote cache setup. The process implies good knowledge of both Maven and the
-project. Due to Maven model limitation the process is semi-manual, but allows you to achieve sufficient control and
-transparency over caching logic.
+This document describes a generic approach to remote cache setup. Due to Maven model limitations, the process is
+semi-manual but allows you to achieve sufficient control and transparency over caching logic. The process implies good
+knowledge of both Maven and the project.
### Before you start
-Before you start, please keep in mind basic principles:
+Before you start, please keep in mind the basic principles:
-* Cache is key based, the key is produced by HashTree-like technique. The key is produced by hashing every configured
- source code file, every dependency and effective pom (including plugin parameters). Every element's hash contributes
- to the key. In order to produce the same key there engine must consume exactly the same hashes.
-* There is no built-in normalization of line endings in this implementation, file hash calculation is raw bytes
- based. The most obvious implication could be illustrated by a simple Git checkout. By default, git will check out
- source code with CRLF line endings on win and LF on Linux. Because of that builds over same commit on a Linux agent
- and local build on Windows workstation will yield different hashes.
-* Parameters of plugins are reconciled in runtime. For example to avoid of accidentally reusing builds which never run
- tests ensure that critical surefire parameters are tracked (`skipTests` and similar) in config. The same applies for
- all over plugins.
+* The cache is key based. Using the HashTree-like technique it produces the key by hashing every configured source code
+ file, every dependency, and effective pom (including plugin parameters). Every element's hash contributes to the key.
+ The engine must consume identical hashes to produce the same key.
+* There is no built-in normalization of line endings in this implementation. File hash calculation is raw bytes based. A
+ common pitfall is a Git checkout with CRLF on Win and LF on Linux. The same commit with different line endings will
+ produce different hashes.
+* Parameters of plugins are reconciled in runtime. For example, to avoid accidentally reusing builds that never run
+ tests, track all critical surefire parameters (`skipTests` and similar) in config. The same applies to all other
+ plugins.
## Step-By-Step
-### Minimize number of moving parts
+### Minimize the number of moving parts
-* Run build with single threaded builder to make sure logs from different modules do not interfere
-* Use the same branch which no-one else commits to
-* Designate single agent/node for CI builds
+* Run build with a single-threaded builder to make sure logs from different modules do not interfere
+* Use the same branch which no one else commits to
+* Designate a single agent/node for CI builds
* Preferably use the same OS between CI and local machine
### Fork branch for cache setup purposes
-Fork stable code branch for cache setup purposes as you will need source code which doesn't change over time of setup.
-Also, you likely will need to do code changes as long as you go.
+Fork stable code branch for cache setup purposes, as you will need source code that doesn't change over setup time.
+Also, you likely need to make code changes as long as you go.
-### Setup http server to store artifacts
+### Setup HTTP server to store artifacts
-In order to share build results cache needs a shared storage. The simplest option is to set up a http server which
-supports http PUT/GET/HEAD operations will suffice (Nginx, Apache or similar). Add the url to config and
+To share results, the cache needs shared storage. The simplest option is to set up an HTTP server that supports HTTP
+PUT/GET/HEAD operations (Nginx, Apache, or similar). Add the URL to the config and
change `remote@enabled` to true:
```xml
-<remote enabled="true">
+<!--Default @id is 'cache'-->
+<remote enabled="true" id="my-cache">
<url>http://your-buildcache-url</url>
</remote>
```
-If proxy or authentication is required to access remote cache, add server record to settings.xml as described
-in [Servers](https://maven.apache.org/settings.html#Servers). The server should be referenced from cache config:
+If proxy or authentication is required to access the remote cache, add server record to settings.xml as described
+in [Servers](https://maven.apache.org/settings.html#Servers). Reference the server in the cache config:
-```
-TBD
+```xml
+
+<servers>
+ <server>
+ <!-- Should match id attribute from the 'remote' tag in cache config or should be 'cache' by default -->
+ <id>my-cache</id>
+ <username>[cache user]</username>
+ <password>[cache password]</password>
+ </server>
+</servers>
```
-Beside the http server, remote cache could be configured using any storage which is supported by [Maven Resolver](https://maven.apache.org/resolver/). That includes a wide set of options, including SSH, FTP and many others through the use of [Maven Wagon](https://maven.apache.org/wagon/). See Wagon documentation for a full list of options and other details.
+Besides the HTTP server, the remote cache could be configured using any storage supported
+by [Maven Resolver](https://maven.apache.org/resolver/). That includes a broad set of options, including SSH, FTP, and
+many others, using [Maven Wagon](https://maven.apache.org/wagon/). See Wagon documentation for a complete list of
+options and other details.
### Build selection
-Build stored in cache ideally should be a build assembled in the most correct, comprehensive and complete way. Pull
-requests builds are good candidates to populate cache usually because this is there quality safeguards are applied
-normally.
+Build stored in cache ideally should be assembled in the most correct, comprehensive, and complete way. Pull request
+builds are typically good candidates to populate the cache because this is where quality safeguards are commonly
+applied.
### CI Build setup to seed shared cache
-Allow writes in remote cache add jvm property to designated CI builds.
+To allow writes in the remote cache, add JVM property to designated CI builds.
```
-Dmaven.build.cache.remote.save.enabled=true
```
-Run the build, review log and ensure that artifacts are uploaded to remote cache. Now, rerun build and ensure that it
+Run the build, review the log, and verify the successful upload of cache artifacts to the remote cache. Now, rerun the
+build and ensure that it
completes almost instantly because it is fully cached.
-### Remote cache relocation to local builds
+### Make remote cache portable
-As practice shows, developers often don't realize that builds they run in local and CI environments are different. So
-straightforward attempt to reuse remote cache in local build usually results in cache misses because of difference in
-plugins, parameters, profiles, environment, etc. In order to reuse results you might need to change poms, cache config,
-CI jobs and the project itself. This part is usually most challenging and time-consuming. Follow steps below to
-iteratively achieve working configuration.
+As practice shows, builds that run on local workstations and CI environments differ. A straightforward attempt to
+reuse cache produced in different environments usually results in cache misses. Out-of-the-box caches are rarely
+portable because of differences in CI jobs, environment specifics, and project configurations. Making the cache portable
+is typically the most challenging and time-consuming part of the setup. Follow the steps below to achieve a portable
+configuration iteratively.
-* Enable fail fast mode to fail build on the first discrepancy between
-* Provide reference to the CI build as a baseline for comparison between your local and remote builds. Go to the
- reference CI build log and one of the final lines of the build should be a line about saving `build-cache-report.xml`
+* Enable fail-fast mode to fail the build on the first discrepancy between the remote and local data
+* Provide reference to the CI build as a baseline for comparing your local and remote builds. Go to the reference CI
+ build log, and in the end, find the line about saving `build-cache-report.xml`:
```
[INFO] [CACHE] Saved to remote cache https://your-cache-url/<...>/915296a3-4596-4eb5-bf37-f6e13ebe087e/build-cache-report.xml
@@ -106,13 +118,13 @@ iteratively achieve working configuration.
Copy the link to a `build-cache-report.xml` and provide it to your local build as a baseline for comparison.
-* Run local build. Command line should look similar to this:
+* Run local build. The command line should look similar to this:
```bash
mvn verify -Dmaven.build.cache.failFast=true -Dmaven.build.cache.baselineUrl=https://your-cache-url/<...>/915296a3-4596-4eb5-bf37-f6e13ebe087e/build-cache-report.xml
```
-Once discrepancy between remote and local builds detected cache will fail with diagnostic info in
+Once a discrepancy between remote and local builds is detected, the cache will fail and write report in the
project's `target/incremental-maven` directory:
```
@@ -122,8 +134,9 @@ project's `target/incremental-maven` directory:
```
Review `buildsdiff.xml` file and eliminate detected discrepancies. You can also diff build-info files directly to get
-low level insights. See techniques to configure cache in [How-To](how-to.md) and troubleshooting of typical issues
-in the section below.
+low-level insights. See techniques to configure cache in [How-To](how-to.md) and troubleshooting of typical issues
+in the section below. Also, it is possible to diff remote and local `buildInfo.xml` files directly using any tool of
+your preference.
If you are using a webdav remote server (which is the most easiest to use to create directories on a remote server), from Maven 3.9.1 and 4.0.0-alpha5, support of webdav has been removed per default.
You need to use the following extra configuration:
@@ -142,76 +155,80 @@ Or configure your remote with the following configuration (`dav:url`)
## Common issues
-### Issue 1: Local checkout is with different line endings
+### Issue 1: Local checkout with different line endings
-Solution: normalise line endings. Current implementation doesn't have built-in line endings normalization, it has to be
-done externally. In git it is recommended to use `.gitattributes` file to establish consistent line endings across all
-envs for file types specific to this project
+Solution: normalize line endings. The current implementation doesn't have built-in line endings normalization, need to
+apply it manually. In git, using the `.gitattributes` file is the recommended way to establish consistent line endings
+across all environments.
### Issue 2: Effective poms mismatch because of plugins injection by profiles
-Different profiles between remote and local builds likely result in different text of effective poms. As effective pom
-contributes hash value to the key that could lead to cache misses. Solution: instead of adding/removing specific plugins
-by profiles, set default value of the plugin's `skip` or `disabled` flag in a profile properties instead. Instead of:
+Different profiles between remote and local builds are likely to result in different content of effective poms.
+Effective pom contributes hash value to the key that could lead to cache misses. Solution: instead of adding/removing
+specific plugins by profiles, set the default value of the plugin's `skip` or `disabled` flag in profile properties
+instead. Instead of:
```xml
+
<profiles>
- <profile>
- <id>run-plugin-in-ci-only</id>
- <build>
- <plugins>
- <plugin>
- <artifactId>surefire-report-maven-plugin</artifactId>
- <configuration>
- <!-- my configuration -->
- </configuration>
- </plugin>
- </plugins>
- </build>
- </profile>
+ <profile>
+ <id>run-plugin-in-ci-only</id>
+ <build>
+ <plugins>
+ <plugin>
+ <artifactId>surefire-report-maven-plugin</artifactId>
+ <configuration>
+ <!-- my configuration -->
+ </configuration>
+ </plugin>
+ </plugins>
+ </build>
+ </profile>
</profiles>
```
Use:
```xml
+
<properties>
<!-- default value -->
<skip.plugin.property>true</skip.plugin.property>
</properties>
<build>
- <plugins>
- <plugin>
- <artifactId>maven-surefire-plugin</artifactId>
- <configuration>
- <!-- plugin behavior is controlled by property -->
- <skip>${skip.plugin.property}</skip>
- </configuration>
- </plugin>
- </plugins>
+<plugins>
+ <plugin>
+ <artifactId>maven-surefire-plugin</artifactId>
+ <configuration>
+ <!-- flag to control the plugin behavior - skip or run -->
+ <skip>${skip.plugin.property}</skip>
+ </configuration>
+ </plugin>
+</plugins>
</build>
<profiles>
- <profile>
- <id>run-plugin-in-ci-only</id>
- <properties>
- <!-- override to run plugin in reference ci build -->
- <skip.plugin.property>false</skip.plugin.property>
- </properties>
- </profile>
+<profile>
+ <id>run-plugin-in-ci-only</id>
+ <properties>
+ <!-- override to run the plugin in reference ci build -->
+ <skip.plugin.property>false</skip.plugin.property>
+ </properties>
+</profile>
</profiles>
```
-Hint: effective poms could be found in `buildinfo` files under `/build/projectsInputInfo/item[@type='pom']`
-xpath (`item type="pom"`).
+Check effective pom in `buildinfo` files under `/build/projectsInputInfo/item[@type='pom']` xpath to see the content.
-### Issue 3: Effective pom mismatch because of environment specific properties
+### Issue 3: Effective pom mismatch because of environment-specific properties
-Potential reason: Sometimes it is not possible to avoid discrepancies in different environments - for example if plugin
-takes command line as parameter, it will be likely different on Win and linux. Such commands will appear in effective
-pom as a different literal values and will result in a different effective pom hash and cache key mismatch. Solution:
-filter out such properties from effective pom:
+Sometimes it is impossible to avoid discrepancies in different environments - for example, if a plugin
+takes the command line as a parameter, it likely spells differently on Win and Linux. Such commands will appear in the
+effective
+pom as different literal values, resulting in a different effective pom hash and cache key mismatch. Solution:
+Filter out environment-specific properties from the effective pom model:
```xml
+
<input>
<global>
...
@@ -230,20 +247,22 @@ Potential reasons: plugins or tests emit temporary files (logs and similar) in n
global exclusions list to filter out the unexpected files:
```xml
+
<global>
<exclude>tempfile.out</exclude>
</global>
```
-see sample config for exact syntax
+See the sample config for the exact syntax.
### Issue 5: Difference in tracked plugin properties
-Tracked property in config means it is critical for determining is build up to date or not. Discrepancies could happen
-for any plugin for a number of reasons. Example: local build is using java target 1.6, remote: 1.8. `buildsdiff.xml`
-will produce something like
+Tracked properties ensure consistent values with the cache record. Discrepancies could happen
+for any plugin for many reasons. Example: local build is using java target 1.6, remote: 1.8. `buildsdiff.xml` will
+produce something like
```xml
+
<mismatch item="target"
current="1.8"
baseline="1.6"
@@ -251,5 +270,5 @@ will produce something like
resolution="Align properties between remote and local build or remove property from tracked list if mismatch could be tolerated. In some cases it is possible to add skip value to ignore lax mismatch"/>
```
-Solution is at your discretion. If the property is tracked, out-of-date status is fair and expected. If you want to
+The solution is at your discretion. For a tracked property, out-of-date status is fair and expected. If you want to
relax consistency rules in favor of compatibility, remove property from the reconciliations list
diff --git a/src/site/markdown/usage.md b/src/site/markdown/usage.md
index 88145bc..92ef341 100644
--- a/src/site/markdown/usage.md
+++ b/src/site/markdown/usage.md
@@ -17,43 +17,48 @@
## Normal usage
-Once extension is activated, cache will kick-in automatically on every lifecycle build of phase `package` or higher.
+Once the extension is activated, the cache automatically kicks in on every `package` or higher phase.
## Subtree builds
-Build could be invoked on any module in project and will try to discover cache by introspecting dependencies. In order
-to identify which dependencies are part of cacheable project the cache engine needs to know:
+The build could be invoked on any module in the project and will try to discover the cache by introspecting
+dependencies. In order
+to identify which dependencies are cacheable, the cache introspects the following:
-* Full project root location which must be passed with `-Dmaven.multiModuleProjectDirectory`
-* Profiles which activate full graph in project build. Underlying implementation logic is to introspect reactor
- graph in `full` mode to discover which dependencies are part of the full project and cacheable. This information will
- be used when subtree is being build to identify dependencies as a part of a wider project and process them from cache:
+* Project root location passed with `-Dmaven.multiModuleProjectDirectory`
+* Profiles that activate full graph in the project build. The cache uses this `full` mode to categorize project modules
+ as cacheable or not:
```xml
+
<configuration>
- ...
- <multiModule>
- <discovery>
- <scanProfiles>
- <scanProfile>my-full-project-profile</scanProfile>
- </scanProfiles>
- </discovery>
- </multiModule>
- ...
+ ...
+ <multiModule>
+ <discovery>
+ <scanProfiles>
+ <scanProfile>my-full-project-profile</scanProfile>
+ </scanProfiles>
+ </discovery>
+ </multiModule>
+ ...
</configuration>
```
## Disable cache
Disable in config:
+
```xml
+
<cache>
<configuration>
<enabled>false</enabled>
</configuration>
</cache>
```
-On command line:
+
+On the command line:
+
```
-Dmaven.build.cache.enabled=false
```
@@ -67,8 +72,8 @@ When a configuration is disabled by default in the config, it can be enabled via
Build cache extension is generally compatible with IDEs with one limitation:
-* The cache doesn't restore full project state. Compiled classes and other output directories will not be restored from
- cache if `clean` was invoked. In order to work efficiently, IDE should be configured to not use Maven
- output (`target`) directories for compilation. In that case compilation caches will be maintained by IDE leveraging
- both fast builds and fast compilation
-
+* The cache doesn't restore the entire project state. Compiled classes, unpacked artifacts, and similar ones typically
+ will not be restored in the build directory (aka `target`). Configure your IDE to not use Maven
+ build (`target`) directories for compilation and execution. In that case, IDE will provide fast compilation using
+ native caches, and
+ the build cache will supplement that with fast builds.