[spark] branch master updated: [SPARK-32556][INFRA] Fix release script to have urlencoded passwords where required

[pr...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

prashant pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new 6c3d0a4  [SPARK-32556][INFRA] Fix release script to have urlencoded passwords where required
6c3d0a4 is described below

commit 6c3d0a44056cc5ca1d304b5a8a03d2f02974e58b
Author: Prashant Sharma <pr...@apache.org>
AuthorDate: Fri Aug 7 03:31:45 2020 -0500

    [SPARK-32556][INFRA] Fix release script to have urlencoded passwords where required
    
    ### What changes were proposed in this pull request?
    1. URL encode the `ASF_PASSWORD` of the release manager.
    2. Update the image to install `qpdf` and `jq` dep
    3. Increase the JVM HEAM memory for maven build.
    
    ### Why are the changes needed?
    Release script takes hours to run, and if a single failure happens about somewhere midway, then either one has to get down to manually doing stuff or re run the entire script. (This is my understanding) So, I have made the fixes of a few failures, discovered so far.
    
    1. If the release manager password contains a char, that is not allowed in URL, then it fails the build at the clone spark step.
    `git clone "https://$ASF_USERNAME:$ASF_PASSWORD$ASF_SPARK_REPO" -b $GIT_BRANCH`
    
              ^^^ Fails with bad URL
    
    `ASF_USERNAME` may not be URL encoded, but we need to encode `ASF_PASSWORD`.
    
    ### Does this PR introduce _any_ user-facing change?
    No
    
    ### How was this patch tested?
    By running the release for branch-2.4, using both type of passwords, i.e. passwords with special chars and without it.
    
    Closes #29373 from ScrapCodes/release-script-fix2.
    
    Lead-authored-by: Prashant Sharma <pr...@apache.org>
    Co-authored-by: Prashant Sharma <pr...@in.ibm.com>
    Signed-off-by: Prashant Sharma <pr...@apache.org>
---
 dev/create-release/release-tag.sh      | 6 +++++-
 dev/create-release/release-util.sh     | 2 +-
 dev/create-release/spark-rm/Dockerfile | 2 ++
 3 files changed, 8 insertions(+), 2 deletions(-)

diff --git a/dev/create-release/release-tag.sh b/dev/create-release/release-tag.sh
index 39856a9..a9a518f 100755
--- a/dev/create-release/release-tag.sh
+++ b/dev/create-release/release-tag.sh
@@ -64,8 +64,12 @@ init_maven_sbt
 
 ASF_SPARK_REPO="gitbox.apache.org/repos/asf/spark.git"
 
+function uriencode { jq -nSRr --arg v "$1" '$v|@uri'; }
+
+declare -r ENCODED_ASF_PASSWORD=$(uriencode "$ASF_PASSWORD")
+
 rm -rf spark
-git clone "https://$ASF_USERNAME:$ASF_PASSWORD@$ASF_SPARK_REPO" -b $GIT_BRANCH
+git clone "https://$ASF_USERNAME:$ENCODED_ASF_PASSWORD@$ASF_SPARK_REPO" -b $GIT_BRANCH
 cd spark
 
 git config user.name "$GIT_NAME"
diff --git a/dev/create-release/release-util.sh b/dev/create-release/release-util.sh
index af9ed20..7961eed 100755
--- a/dev/create-release/release-util.sh
+++ b/dev/create-release/release-util.sh
@@ -228,7 +228,7 @@ function init_maven_sbt {
   if [[ $JAVA_VERSION < "1.8." ]]; then
     # Needed for maven central when using Java 7.
     SBT_OPTS="-Dhttps.protocols=TLSv1.1,TLSv1.2"
-    MVN_EXTRA_OPTS="-Dhttps.protocols=TLSv1.1,TLSv1.2"
+    MVN_EXTRA_OPTS="-Xmx2g -XX:ReservedCodeCacheSize=1g -Dhttps.protocols=TLSv1.1,TLSv1.2"
     MVN="$MVN $MVN_EXTRA_OPTS"
   fi
   export MVN MVN_EXTRA_OPTS SBT_OPTS
diff --git a/dev/create-release/spark-rm/Dockerfile b/dev/create-release/spark-rm/Dockerfile
index a02a6b7..200f8b9 100644
--- a/dev/create-release/spark-rm/Dockerfile
+++ b/dev/create-release/spark-rm/Dockerfile
@@ -66,6 +66,8 @@ RUN apt-get clean && apt-get update && $APT_INSTALL gnupg ca-certificates && \
   $APT_INSTALL nodejs && \
   # Install needed python packages. Use pip for installing packages (for consistency).
   $APT_INSTALL libpython3-dev python3-pip python3-setuptools && \
+  # qpdf is required for CRAN checks to pass.
+  $APT_INSTALL qpdf jq && \
   # Change default python version to python3.
   update-alternatives --install /usr/bin/python python /usr/bin/python2.7 1 && \
   update-alternatives --install /usr/bin/python python /usr/bin/python3.6 2 && \


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org