You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2020/01/20 01:43:53 UTC
[spark] branch branch-2.4 updated: [SPARK-30572][BUILD] Add a
fallback Maven repository
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-2.4
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-2.4 by this push:
new ebaa6fe [SPARK-30572][BUILD] Add a fallback Maven repository
ebaa6fe is described below
commit ebaa6fe1dbefe003a01e7760dbbb643c937c74a0
Author: Dongjoon Hyun <dh...@apple.com>
AuthorDate: Sun Jan 19 17:42:34 2020 -0800
[SPARK-30572][BUILD] Add a fallback Maven repository
### What changes were proposed in this pull request?
This PR aims to add a fallback Maven repository when a mirror to `central` fail.
### Why are the changes needed?
We use `Google Maven Central` in GitHub Action as a mirror of `central`.
However, `Google Maven Central` sometimes doesn't have newly published artifacts
and there is no guarantee when we get the newly published artifacts.
By duplicating `Maven Central` with a new ID, we can add a fallback Maven repository
which is not mirrored by `Google Maven Central`.
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
Manually testing with the new `Twitter` chill artifacts by switching `chill.version` from `0.9.3` to `0.9.5`.
```
$ rm -rf ~/.m2/repository/com/twitter/chill*
$ mvn compile | grep chill
Downloading from google-maven-central: https://maven-central.storage-download.googleapis.com/repos/central/data/com/twitter/chill_2.12/0.9.5/chill_2.12-0.9.5.pom
Downloading from central_without_mirror: https://repo.maven.apache.org/maven2/com/twitter/chill_2.12/0.9.5/chill_2.12-0.9.5.pom
Downloaded from central_without_mirror: https://repo.maven.apache.org/maven2/com/twitter/chill_2.12/0.9.5/chill_2.12-0.9.5.pom (2.8 kB at 11 kB/s)
```
Closes #27281 from dongjoon-hyun/SPARK-30572.
Authored-by: Dongjoon Hyun <dh...@apple.com>
Signed-off-by: Dongjoon Hyun <dh...@apple.com>
(cherry picked from commit c992716a33d78ce0a9aa78b26f5bdb45c26c2a01)
Signed-off-by: Dongjoon Hyun <dh...@apple.com>
---
.github/workflows/branch-2.4.yml | 7 +++++++
pom.xml | 16 ++++++++++++++++
2 files changed, 23 insertions(+)
diff --git a/.github/workflows/branch-2.4.yml b/.github/workflows/branch-2.4.yml
index 2c734f9..85b456f 100644
--- a/.github/workflows/branch-2.4.yml
+++ b/.github/workflows/branch-2.4.yml
@@ -51,6 +51,13 @@ jobs:
export MAVEN_OPTS="-Xmx2g -XX:ReservedCodeCacheSize=512m -Dorg.slf4j.simpleLogger.defaultLogLevel=WARN"
export MAVEN_CLI_OPTS="--no-transfer-progress"
mkdir -p ~/.m2
+ # `Maven Central` is too flaky in terms of downloading artifacts in `GitHub Action` environment.
+ # `Google Maven Central Mirror` is too slow in terms of sycing upstream. To get the best combination,
+ # 1) we set `Google Maven Central` as a mirror of `central` in `GitHub Action` environment only.
+ # 2) we duplicates `Maven Central` in pom.xml with ID `central_without_mirror`.
+ # In other words, in GitHub Action environment, `central` is mirrored by `Google Maven Central` first.
+ # If `Google Maven Central` doesn't provide the artifact due to its slowness, `central_without_mirror` will be used.
+ # Note that we aim to achieve the above while keeping the existing behavior of non-`GitHub Action` environment unchanged.
echo "<settings><mirrors><mirror><id>google-maven-central</id><name>GCS Maven Central mirror</name><url>https://maven-central.storage-download.googleapis.com/repos/central/data/</url><mirrorOf>central</mirrorOf></mirror></mirrors></settings>" > ~/.m2/settings.xml
./build/mvn $MAVEN_CLI_OPTS -DskipTests -Pyarn -Pmesos -Pkubernetes -Phive -Phive-thriftserver -Pscala-${{ matrix.scala }} -P${{ matrix.hadoop }} -Phadoop-cloud install
rm -rf ~/.m2/repository/org/apache/spark
diff --git a/pom.xml b/pom.xml
index 1713df7..f518386 100644
--- a/pom.xml
+++ b/pom.xml
@@ -242,6 +242,22 @@
<enabled>false</enabled>
</snapshots>
</repository>
+ <repository>
+ <id>central_without_mirror</id>
+ <!--
+ This is used as a fallback when a mirror to `central` fail.
+ For example, when we use Google Maven Central in GitHub Action as a mirror of `central`,
+ this will be used when Google Maven Central is out of sync due to its late sync cycle.
+ -->
+ <name>Maven Repository</name>
+ <url>https://repo.maven.apache.org/maven2</url>
+ <releases>
+ <enabled>true</enabled>
+ </releases>
+ <snapshots>
+ <enabled>false</enabled>
+ </snapshots>
+ </repository>
</repositories>
<pluginRepositories>
<pluginRepository>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org