(spark) branch master updated: [SPARK-48540][CORE] Avoid ivy output loading settings to stdout

[ya...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

yao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new f4434c36cc4f [SPARK-48540][CORE] Avoid ivy output loading settings to stdout
f4434c36cc4f is described below

commit f4434c36cc4f7b0147e0e8fe26ac0f177a5199cd
Author: sychen <sy...@ctrip.com>
AuthorDate: Thu Jun 6 14:35:52 2024 +0800

    [SPARK-48540][CORE] Avoid ivy output loading settings to stdout
    
    ### What changes were proposed in this pull request?
    This PR aims to avoid ivy output loading settings to stdout.
    
    ### Why are the changes needed?
    Now `org.apache.spark.util.MavenUtils#getModuleDescriptor` will output the following string to stdout.
    
    This is due to the modified code order in SPARK-32596 .
    
    ```
    :: loading settings :: url = jar:file:xxxx/jars/ivy-2.5.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
    ```
    
    Stack trace
    ```java
            at org.apache.ivy.core.settings.IvySettings.load(IvySettings.java:404)
            at org.apache.ivy.core.settings.IvySettings.loadDefault(IvySettings.java:443)
            at org.apache.ivy.Ivy.configureDefault(Ivy.java:435)
            at org.apache.ivy.core.IvyContext.getDefaultIvy(IvyContext.java:201)
            at org.apache.ivy.core.IvyContext.getIvy(IvyContext.java:180)
            at org.apache.ivy.core.IvyContext.getSettings(IvyContext.java:216)
            at org.apache.ivy.core.module.status.StatusManager.getCurrent(StatusManager.java:40)
            at org.apache.ivy.core.module.descriptor.DefaultModuleDescriptor.<init>(DefaultModuleDescriptor.java:206)
            at org.apache.ivy.core.module.descriptor.DefaultModuleDescriptor.newDefaultInstance(DefaultModuleDescriptor.java:107)
            at org.apache.ivy.core.module.descriptor.DefaultModuleDescriptor.newDefaultInstance(DefaultModuleDescriptor.java:66)
            at org.apache.spark.deploy.SparkSubmitUtils$.getModuleDescriptor(SparkSubmit.scala:1413)
            at org.apache.spark.deploy.SparkSubmitUtils$.resolveMavenCoordinates(SparkSubmit.scala:1460)
            at org.apache.spark.util.DependencyUtils$.resolveMavenDependencies(DependencyUtils.scala:185)
            at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:327)
            at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:942)
            at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:181)
    ```
    
    ### Does this PR introduce _any_ user-facing change?
    No
    
    ### How was this patch tested?
    local test
    
    ### Was this patch authored or co-authored using generative AI tooling?
    No
    
    Closes #46882 from cxzl25/SPARK-48540.
    
    Authored-by: sychen <sy...@ctrip.com>
    Signed-off-by: Kent Yao <ya...@apache.org>
---
 .../src/main/scala/org/apache/spark/util/MavenUtils.scala   | 13 +++++++------
 1 file changed, 7 insertions(+), 6 deletions(-)

diff --git a/common/utils/src/main/scala/org/apache/spark/util/MavenUtils.scala b/common/utils/src/main/scala/org/apache/spark/util/MavenUtils.scala
index 08291859a32c..ae00987cd69f 100644
--- a/common/utils/src/main/scala/org/apache/spark/util/MavenUtils.scala
+++ b/common/utils/src/main/scala/org/apache/spark/util/MavenUtils.scala
@@ -462,14 +462,13 @@ private[spark] object MavenUtils extends Logging {
       val sysOut = System.out
       // Default configuration name for ivy
       val ivyConfName = "default"
-
-      // A Module descriptor must be specified. Entries are dummy strings
-      val md = getModuleDescriptor
-
-      md.setDefaultConf(ivyConfName)
+      var md: DefaultModuleDescriptor = null
       try {
         // To prevent ivy from logging to system out
         System.setOut(printStream)
+        // A Module descriptor must be specified. Entries are dummy strings
+        md = getModuleDescriptor
+        md.setDefaultConf(ivyConfName)
         val artifacts = extractMavenCoordinates(coordinates)
         // Directories for caching downloads through ivy and storing the jars when maven coordinates
         // are supplied to spark-submit
@@ -548,7 +547,9 @@ private[spark] object MavenUtils extends Logging {
         }
       } finally {
         System.setOut(sysOut)
-        clearIvyResolutionFiles(md.getModuleRevisionId, ivySettings.getDefaultCache, ivyConfName)
+        if (md != null) {
+          clearIvyResolutionFiles(md.getModuleRevisionId, ivySettings.getDefaultCache, ivyConfName)
+        }
       }
     }
   }


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org