[spark] branch master updated: [SPARK-43461][BUILD] Skip compiling useless files when making distribution

[ya...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

yangjie01 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new 772ef41d6a4 [SPARK-43461][BUILD] Skip compiling useless files when making distribution
772ef41d6a4 is described below

commit 772ef41d6a40ed219bcdf8a51691391fa6eba75d
Author: Yuming Wang <yu...@ebay.com>
AuthorDate: Wed May 17 11:33:26 2023 +0800

    [SPARK-43461][BUILD] Skip compiling useless files when making distribution
    
    ### What changes were proposed in this pull request?
    
    This PR add more skip properties when making distribution:
    - `-Dmaven.javadoc.skip=true` to skip generating javadoc
    - `-Dmaven.scaladoc.skip=true` to skip generating scaladoc. Please see: https://davidb.github.io/scala-maven-plugin/doc-jar-mojo.html#skip
    - `-Dmaven.source.skip` to skip generating sources.jar
    - `-Dcyclonedx.skip=true` to skip making bom. Please see: https://cyclonedx.github.io/cyclonedx-maven-plugin/makeBom-mojo.html#skip
    
    ### Why are the changes needed?
    
    Reduce time spent on making distribution.
    
    ### Does this PR introduce _any_ user-facing change?
    
    No.
    
    ### How was this patch tested?
    
    Manual test:
    ```sh
    ./dev/make-distribution.sh --tgz --pip -Phadoop-3 -Phive -Phive-thriftserver -Pyarn -Phadoop-provided
    ```
    
    Before this PR | After this PR
    -- | --
    43 min total from scheduled to completion | 23 min total from scheduled to completion
    
    Closes #41141 from wangyum/SPARK-43461.
    
    Lead-authored-by: Yuming Wang <yu...@ebay.com>
    Co-authored-by: Yuming Wang <wg...@gmail.com>
    Signed-off-by: yangjie01 <ya...@baidu.com>
---
 dev/make-distribution.sh | 8 +++++++-
 pom.xml                  | 2 ++
 2 files changed, 9 insertions(+), 1 deletion(-)

diff --git a/dev/make-distribution.sh b/dev/make-distribution.sh
index 948ee19fbac..ef7c010e930 100755
--- a/dev/make-distribution.sh
+++ b/dev/make-distribution.sh
@@ -166,7 +166,13 @@ export MAVEN_OPTS="${MAVEN_OPTS:--Xss128m -Xmx4g -XX:ReservedCodeCacheSize=128m}
 # Store the command as an array because $MVN variable might have spaces in it.
 # Normal quoting tricks don't work.
 # See: http://mywiki.wooledge.org/BashFAQ/050
-BUILD_COMMAND=("$MVN" clean package -DskipTests $@)
+BUILD_COMMAND=("$MVN" clean package \
+    -DskipTests \
+    -Dmaven.javadoc.skip=true \
+    -Dmaven.scaladoc.skip=true \
+    -Dmaven.source.skip \
+    -Dcyclonedx.skip=true \
+    $@)
 
 # Actually build the jar
 echo -e "\nBuilding with..."
diff --git a/pom.xml b/pom.xml
index 1f6305b77ab..7f37f8f1a3a 100644
--- a/pom.xml
+++ b/pom.xml
@@ -175,6 +175,7 @@
       See: SPARK-36547, SPARK-38394.
        -->
     <scala-maven-plugin.version>4.8.0</scala-maven-plugin.version>
+    <maven.scaladoc.skip>false</maven.scaladoc.skip>
     <versions-maven-plugin.version>2.15.0</versions-maven-plugin.version>
     <!-- for now, not running scalafmt as part of default verify pipeline -->
     <scalafmt.skip>true</scalafmt.skip>
@@ -2851,6 +2852,7 @@
             <checkMultipleScalaVersions>true</checkMultipleScalaVersions>
             <failOnMultipleScalaVersions>true</failOnMultipleScalaVersions>
             <recompileMode>incremental</recompileMode>
+            <skip>${maven.scaladoc.skip}</skip>
             <args>
               <arg>-unchecked</arg>
               <arg>-deprecation</arg>


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org