You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@kyuubi.apache.org by "turboFei (via GitHub)" <gi...@apache.org> on 2024/04/01 20:09:13 UTC

[PR] Scala repl urls [kyuubi]

turboFei opened a new pull request, #6235:
URL: https://github.com/apache/kyuubi/pull/6235

   # :mag: Description
   ## Issue References ๐Ÿ”—
   <!-- Append the issue number after #. If there is no issue for you to link create one or -->
   <!-- If there are no issues to link, please provide details here. -->
   
   This pull request fixes #
   
   ## Describe Your Solution ๐Ÿ”ง
   
   Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
   
   
   ## Types of changes :bookmark:
   <!--- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
   - [ ] Bugfix (non-breaking change which fixes an issue)
   - [ ] New feature (non-breaking change which adds functionality)
   - [ ] Breaking change (fix or feature that would cause existing functionality to change)
   
   ## Test Plan ๐Ÿงช
   
   #### Behavior Without This Pull Request :coffin:
   
   
   #### Behavior With This Pull Request :tada:
   
   
   #### Related Unit Tests
   
   
   ---
   
   # Checklist ๐Ÿ“
   <!--- Go over all the following points, and put an `x` in all the boxes that apply. -->
   <!--- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
   
   - [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
   
   **Be nice. Be informative.**
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


Re: [PR] [KYUUBI #6223] Fix Scala interpreter can not access spark.jars issue [kyuubi]

Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #6235:
URL: https://github.com/apache/kyuubi/pull/6235#issuecomment-2031143415

   I verified this patch with Spark(3.3) on K8s cluster mode.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


Re: [PR] Fix Scala interpreter can not access spark.jars issue [kyuubi]

Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #6235:
URL: https://github.com/apache/kyuubi/pull/6235#issuecomment-2031010587

   cc @cxzl25 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


Re: [PR] Fix Scala interpreter can not access spark.jars issue [kyuubi]

Posted by "turboFei (via GitHub)" <gi...@apache.org>.
turboFei commented on code in PR #6235:
URL: https://github.com/apache/kyuubi/pull/6235#discussion_r1547086187


##########
externals/kyuubi-spark-sql-engine/src/main/scala-2.12/org/apache/kyuubi/engine/spark/repl/KyuubiSparkILoop.scala:
##########
@@ -28,47 +29,35 @@ import org.apache.spark.repl.SparkILoop
 import org.apache.spark.sql.{DataFrame, SparkSession}
 import org.apache.spark.util.MutableURLClassLoader
 
-import org.apache.kyuubi.Utils
+import org.apache.kyuubi.{Logging, Utils}
 
 private[spark] case class KyuubiSparkILoop private (
     spark: SparkSession,
     output: ByteArrayOutputStream)
-  extends SparkILoop(None, new PrintWriter(output)) {
+  extends SparkILoop(None, new PrintWriter(output)) with Logging {
   import KyuubiSparkILoop._
 
   val result = new DataFrameHolder(spark)
 
   private def initialize(): Unit = withLockRequired {
+    val currentClassLoader = Thread.currentThread().getContextClassLoader
+    val allJars = getAllJars(currentClassLoader)
+    info(s"Adding jars to Scala interpreter's class path: ${allJars.mkString(File.pathSeparator)}")
     settings = new Settings
     val interpArguments = List(
       "-Yrepl-class-based",
       "-Yrepl-outdir",
-      s"${spark.sparkContext.getConf.get("spark.repl.class.outputDir")}")
+      s"${spark.sparkContext.getConf.get("spark.repl.class.outputDir")}",
+      "-classpath",
+      allJars.mkString(File.pathSeparator))

Review Comment:
   Tested with yarn-cluster mode:
   ```
   24/04/01 20:12:30 INFO KyuubiSparkILoop: Adding jars to Scala interpreter's class path: file:/hadoop/7/yarn/local/usercache/b_stf/appcache/application_1710906451321_1347246/container_e4072_1710906451321_1347246_01_000002/__app__.jar:file:/hadoop/7/yarn/local/usercache/b_stf/appcache/application_1710906451321_1347246/container_e4072_1710906451321_1347246_01_000002/delta-core_2.12-1.0.2.0.3.0.jar:file:/hadoop/7/yarn/local/usercache/b_stf/appcache/application_1710906451321_1347246/container_e4072_1710906451321_1347246_01_000002/iceberg-spark-runtime-3.1_2.12-1.2.1.0.7.1.jar:file:/hadoop/7/yarn/local/usercache/b_stf/appcache/application_1710906451321_1347246/container_e4072_1710906451321_1347246_01_000002/simplark-etl-assembly-latest-testing.jar:file:/hadoop/7/yarn/local/usercache/b_stf/appcache/application_1710906451321_1347246/container_e4072_1710906451321_1347246_01_000002/zeta-serde-0.0.4-jar-with-dependencies.jar
   
   ```
   
   ```
   24/04/01 20:12:36 INFO ExecuteScala: Processing b_stf's query[874507af-4dfe-4521-ad26-3caeaa154d28]: PENDING_STATE -> RUNNING_STATE, statement:
   import org.apache.iceberg._
   24/04/01 20:12:36 INFO OperationAuditLogger: operation=874507af-4dfe-4521-ad26-3caeaa154d28	opType=ExecuteScala	state=RUNNING	user=b_stf	session=973b28ec-52d8-454e-b2f1-45fc95957615
   24/04/01 20:12:36 INFO ExecuteScala:
              Spark application name: kyuubi_apollorno_CONNECTION_SPARK_SQL_b_stf_973b28ec-52d8-454e-b2f1-45fc95957615
                    application ID: application_1710906451321_1347246
                    application web UI:  ......
                    master: yarn
                    deploy mode: cluster
                    version: 3.1.1.1.3.0
              Start time: 2024-04-01T20:11:29.244
              User: b_stf
   24/04/01 20:12:36 INFO ExecuteScala: Processing b_stf's query[874507af-4dfe-4521-ad26-3caeaa154d28]: RUNNING_STATE -> FINISHED_STATE, time taken: 0.154 seconds
   24/04/01 20:12:36 INFO OperationAuditLogger: operation=874507af-4dfe-4521-ad26-3caeaa154d28	opType=ExecuteScala	state=FINISHED	user=b_stf	session=973b28ec-52d8-454e-b2f1-45fc95957615
   24/04/01 20:12:36 INFO ExecuteScala: statementId=874507af-4dfe-4521-ad26-3caeaa154d28, operationRunTime=0 ms, operationCpuTime=0 ms
   2024-04-01 20:12:36.439 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[874507af-4dfe-4521-ad26-3caeaa154d28] in FINISHED_STATE
   2024-04-01 20:12:36.439 INFO org.apache.kyuubi.operation.ExecuteStatement: Processing b_stf's query[874507af-4dfe-4521-ad26-3caeaa154d28]: RUNNING_STATE -> FINISHED_STATE, time taken: 0.197 seconds
   +------------------------------+
   |            output            |
   +------------------------------+
   | import org.apache.iceberg._  |
   +------------------------------+
   1 row selected (6.231 seconds)
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


Re: [PR] [KYUUBI #6223] Fix Scala interpreter can not access spark.jars issue [kyuubi]

Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #6235:
URL: https://github.com/apache/kyuubi/pull/6235#issuecomment-2034210003

   Thanks, merged to master/1.9/1.8


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


Re: [PR] Fix Scala interpreter can not access spark.jars issue [kyuubi]

Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #6235:
URL: https://github.com/apache/kyuubi/pull/6235#discussion_r1547091947


##########
externals/kyuubi-spark-sql-engine/src/main/scala-2.12/org/apache/kyuubi/engine/spark/repl/KyuubiSparkILoop.scala:
##########
@@ -28,47 +29,35 @@ import org.apache.spark.repl.SparkILoop
 import org.apache.spark.sql.{DataFrame, SparkSession}
 import org.apache.spark.util.MutableURLClassLoader
 
-import org.apache.kyuubi.Utils
+import org.apache.kyuubi.{Logging, Utils}
 
 private[spark] case class KyuubiSparkILoop private (
     spark: SparkSession,
     output: ByteArrayOutputStream)
-  extends SparkILoop(None, new PrintWriter(output)) {
+  extends SparkILoop(None, new PrintWriter(output)) with Logging {
   import KyuubiSparkILoop._
 
   val result = new DataFrameHolder(spark)
 
   private def initialize(): Unit = withLockRequired {
+    val currentClassLoader = Thread.currentThread().getContextClassLoader
+    val allJars = getAllJars(currentClassLoader)
+    info(s"Adding jars to Scala interpreter's class path: ${allJars.mkString(File.pathSeparator)}")
     settings = new Settings
     val interpArguments = List(
       "-Yrepl-class-based",
       "-Yrepl-outdir",
-      s"${spark.sparkContext.getConf.get("spark.repl.class.outputDir")}")
+      s"${spark.sparkContext.getConf.get("spark.repl.class.outputDir")}",
+      "-classpath",
+      allJars.mkString(File.pathSeparator))
     settings.processArguments(interpArguments, processAll = true)
     settings.usejavacp.value = true
-    val currentClassLoader = Thread.currentThread().getContextClassLoader
     settings.embeddedDefaults(currentClassLoader)
     this.createInterpreter()
     this.initializeSynchronous()
     try {
       this.compilerClasspath
       this.ensureClassLoader()
-      var classLoader: ClassLoader = Thread.currentThread().getContextClassLoader
-      while (classLoader != null) {
-        classLoader match {
-          case loader: MutableURLClassLoader =>
-            val allJars = loader.getURLs.filter { u =>
-              val file = new File(u.getPath)
-              u.getProtocol == "file" && file.isFile &&
-              file.getName.contains("scala-lang_scala-reflect")

Review Comment:
   this actually is a bug, and it should not be an issue when user.classpath.first is false (default)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


Re: [PR] [KYUUBI #6223] Fix Scala interpreter can not access spark.jars issue [kyuubi]

Posted by "turboFei (via GitHub)" <gi...@apache.org>.
turboFei commented on code in PR #6235:
URL: https://github.com/apache/kyuubi/pull/6235#discussion_r1547105582


##########
externals/kyuubi-spark-sql-engine/src/main/scala-2.12/org/apache/kyuubi/engine/spark/repl/KyuubiSparkILoop.scala:
##########
@@ -97,6 +86,23 @@ private[spark] case class KyuubiSparkILoop private (
     }
   }
 
+  private def getAllJars(currentClassLoader: ClassLoader): Array[URL] = {
+    var classLoader: ClassLoader = currentClassLoader
+    var allJars = Array.empty[URL]
+    while (classLoader != null) {
+      classLoader match {
+        case loader: MutableURLClassLoader =>
+          allJars = loader.getURLs.filter { u =>
+            u.getProtocol == "file" && new File(u.getPath).isFile

Review Comment:
   will add comments after the UT finished.
   ```
               // TODO: handle SPARK-47475 since Spark 4.0.0 in the future
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


Re: [PR] [KYUUBI #6223] Fix Scala interpreter can not access spark.jars issue [kyuubi]

Posted by "Yikf (via GitHub)" <gi...@apache.org>.
Yikf commented on code in PR #6235:
URL: https://github.com/apache/kyuubi/pull/6235#discussion_r1547390077


##########
externals/kyuubi-spark-sql-engine/src/main/scala-2.12/org/apache/kyuubi/engine/spark/repl/KyuubiSparkILoop.scala:
##########
@@ -28,47 +29,35 @@ import org.apache.spark.repl.SparkILoop
 import org.apache.spark.sql.{DataFrame, SparkSession}
 import org.apache.spark.util.MutableURLClassLoader
 
-import org.apache.kyuubi.Utils
+import org.apache.kyuubi.{Logging, Utils}
 
 private[spark] case class KyuubiSparkILoop private (
     spark: SparkSession,
     output: ByteArrayOutputStream)
-  extends SparkILoop(None, new PrintWriter(output)) {
+  extends SparkILoop(None, new PrintWriter(output)) with Logging {
   import KyuubiSparkILoop._
 
   val result = new DataFrameHolder(spark)
 
   private def initialize(): Unit = withLockRequired {
+    val currentClassLoader = Thread.currentThread().getContextClassLoader
+    val interpreterClasspath = getAllJars(currentClassLoader).mkString(File.pathSeparator)
+    info(s"Adding jars to Scala interpreter's class path: $interpreterClasspath")
     settings = new Settings
     val interpArguments = List(
       "-Yrepl-class-based",
       "-Yrepl-outdir",
-      s"${spark.sparkContext.getConf.get("spark.repl.class.outputDir")}")
+      s"${spark.sparkContext.getConf.get("spark.repl.class.outputDir")}",
+      "-classpath",
+      interpreterClasspath)
     settings.processArguments(interpArguments, processAll = true)
     settings.usejavacp.value = true
-    val currentClassLoader = Thread.currentThread().getContextClassLoader
     settings.embeddedDefaults(currentClassLoader)
     this.createInterpreter()
     this.initializeSynchronous()
     try {
       this.compilerClasspath
       this.ensureClassLoader()
-      var classLoader: ClassLoader = Thread.currentThread().getContextClassLoader
-      while (classLoader != null) {
-        classLoader match {
-          case loader: MutableURLClassLoader =>
-            val allJars = loader.getURLs.filter { u =>
-              val file = new File(u.getPath)
-              u.getProtocol == "file" && file.isFile &&
-              file.getName.contains("scala-lang_scala-reflect")
-            }
-            this.addUrlsToClassPath(allJars: _*)

Review Comment:
   Just a question, Is there a fundamental difference between `-classpath` and `addUrlsToClassPath `? Why can `-classpath` solve this problem in this way?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


Re: [PR] [KYUUBI #6223] Fix Scala interpreter can not access spark.jars issue [kyuubi]

Posted by "turboFei (via GitHub)" <gi...@apache.org>.
turboFei commented on code in PR #6235:
URL: https://github.com/apache/kyuubi/pull/6235#discussion_r1547111904


##########
externals/kyuubi-spark-sql-engine/src/main/scala-2.12/org/apache/kyuubi/engine/spark/repl/KyuubiSparkILoop.scala:
##########
@@ -97,6 +86,23 @@ private[spark] case class KyuubiSparkILoop private (
     }
   }
 
+  private def getAllJars(currentClassLoader: ClassLoader): Array[URL] = {
+    var classLoader: ClassLoader = currentClassLoader
+    var allJars = Array.empty[URL]
+    while (classLoader != null) {
+      classLoader match {
+        case loader: MutableURLClassLoader =>
+          allJars = loader.getURLs.filter { u =>
+            u.getProtocol == "file" && new File(u.getPath).isFile

Review Comment:
   ok



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


Re: [PR] [KYUUBI #6223] Fix Scala interpreter can not access spark.jars issue [kyuubi]

Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #6235:
URL: https://github.com/apache/kyuubi/pull/6235#discussion_r1547098059


##########
externals/kyuubi-spark-sql-engine/src/main/scala-2.12/org/apache/kyuubi/engine/spark/repl/KyuubiSparkILoop.scala:
##########
@@ -97,6 +86,23 @@ private[spark] case class KyuubiSparkILoop private (
     }
   }
 
+  private def getAllJars(currentClassLoader: ClassLoader): Array[URL] = {
+    var classLoader: ClassLoader = currentClassLoader
+    var allJars = Array.empty[URL]
+    while (classLoader != null) {
+      classLoader match {
+        case loader: MutableURLClassLoader =>
+          allJars = loader.getURLs.filter { u =>
+            u.getProtocol == "file" && new File(u.getPath).isFile

Review Comment:
   it should work well in most cases, but we may need to handle SPARK-47475 in the future



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


Re: [PR] Fix Scala interpreter can not access spark.jars issue [kyuubi]

Posted by "turboFei (via GitHub)" <gi...@apache.org>.
turboFei commented on code in PR #6235:
URL: https://github.com/apache/kyuubi/pull/6235#discussion_r1547050139


##########
externals/kyuubi-spark-sql-engine/src/main/scala-2.12/org/apache/kyuubi/engine/spark/repl/KyuubiSparkILoop.scala:
##########
@@ -60,8 +60,11 @@ private[spark] case class KyuubiSparkILoop private (
             val allJars = loader.getURLs.filter { u =>
               val file = new File(u.getPath)
               u.getProtocol == "file" && file.isFile &&
-              file.getName.contains("scala-lang_scala-reflect")
+              // Some bad spark packages depend on the wrong version of scala-reflect. Blacklist it.
+              !file.getName.contains("scala-lang_scala-reflect")
             }
+            info(s"Adding jars to Scala interpreter's class path: " +
+              allJars.mkString(File.pathSeparator))

Review Comment:
   ```
   24/04/01 19:01:30 INFO KyuubiSparkILoop: Adding jars to Scala interpreter's class path: file:/Users/fwang12/todo/kyuubi/externals/kyuubi-spark-sql-engine/target/kyuubi-spark-sql-engine_2.12-1.10.0-SNAPSHOT.jar:file:/var/folders/pr/9x9rt3j974zfn7p4f5dpf8m80000gq/T/kyuubi-32215772-8004-43c1-a070-1cb0efd79bf0/test-function-02417951-6b76-4cf7-bb4a-5fc20372c034.jar
   
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


Re: [PR] [KYUUBI #6223] Fix Scala interpreter can not access spark.jars issue [kyuubi]

Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 closed pull request #6235: [KYUUBI #6223] Fix Scala interpreter can not access spark.jars issue
URL: https://github.com/apache/kyuubi/pull/6235


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


Re: [PR] [KYUUBI #6223] Fix Scala interpreter can not access spark.jars issue [kyuubi]

Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #6235:
URL: https://github.com/apache/kyuubi/pull/6235#discussion_r1548845718


##########
externals/kyuubi-spark-sql-engine/src/main/scala-2.12/org/apache/kyuubi/engine/spark/repl/KyuubiSparkILoop.scala:
##########
@@ -28,47 +29,35 @@ import org.apache.spark.repl.SparkILoop
 import org.apache.spark.sql.{DataFrame, SparkSession}
 import org.apache.spark.util.MutableURLClassLoader
 
-import org.apache.kyuubi.Utils
+import org.apache.kyuubi.{Logging, Utils}
 
 private[spark] case class KyuubiSparkILoop private (
     spark: SparkSession,
     output: ByteArrayOutputStream)
-  extends SparkILoop(None, new PrintWriter(output)) {
+  extends SparkILoop(None, new PrintWriter(output)) with Logging {
   import KyuubiSparkILoop._
 
   val result = new DataFrameHolder(spark)
 
   private def initialize(): Unit = withLockRequired {
+    val currentClassLoader = Thread.currentThread().getContextClassLoader
+    val interpreterClasspath = getAllJars(currentClassLoader).mkString(File.pathSeparator)
+    info(s"Adding jars to Scala interpreter's class path: $interpreterClasspath")
     settings = new Settings
     val interpArguments = List(
       "-Yrepl-class-based",
       "-Yrepl-outdir",
-      s"${spark.sparkContext.getConf.get("spark.repl.class.outputDir")}")
+      s"${spark.sparkContext.getConf.get("spark.repl.class.outputDir")}",
+      "-classpath",
+      interpreterClasspath)
     settings.processArguments(interpArguments, processAll = true)
     settings.usejavacp.value = true
-    val currentClassLoader = Thread.currentThread().getContextClassLoader
     settings.embeddedDefaults(currentClassLoader)
     this.createInterpreter()
     this.initializeSynchronous()
     try {
       this.compilerClasspath
       this.ensureClassLoader()
-      var classLoader: ClassLoader = Thread.currentThread().getContextClassLoader
-      while (classLoader != null) {
-        classLoader match {
-          case loader: MutableURLClassLoader =>
-            val allJars = loader.getURLs.filter { u =>
-              val file = new File(u.getPath)
-              u.getProtocol == "file" && file.isFile &&
-              file.getName.contains("scala-lang_scala-reflect")
-            }
-            this.addUrlsToClassPath(allJars: _*)

Review Comment:
   Not sure about the underlying implementation, I checked the Spark and Zeppelin code, both use `-classpath`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


Re: [PR] Fix Scala interpreter can not access spark.jars issue [kyuubi]

Posted by "turboFei (via GitHub)" <gi...@apache.org>.
turboFei commented on code in PR #6235:
URL: https://github.com/apache/kyuubi/pull/6235#discussion_r1547092607


##########
externals/kyuubi-spark-sql-engine/src/main/scala-2.12/org/apache/kyuubi/engine/spark/repl/KyuubiSparkILoop.scala:
##########
@@ -28,47 +29,35 @@ import org.apache.spark.repl.SparkILoop
 import org.apache.spark.sql.{DataFrame, SparkSession}
 import org.apache.spark.util.MutableURLClassLoader
 
-import org.apache.kyuubi.Utils
+import org.apache.kyuubi.{Logging, Utils}
 
 private[spark] case class KyuubiSparkILoop private (
     spark: SparkSession,
     output: ByteArrayOutputStream)
-  extends SparkILoop(None, new PrintWriter(output)) {
+  extends SparkILoop(None, new PrintWriter(output)) with Logging {
   import KyuubiSparkILoop._
 
   val result = new DataFrameHolder(spark)
 
   private def initialize(): Unit = withLockRequired {
+    val currentClassLoader = Thread.currentThread().getContextClassLoader
+    val allJars = getAllJars(currentClassLoader)
+    info(s"Adding jars to Scala interpreter's class path: ${allJars.mkString(File.pathSeparator)}")
     settings = new Settings
     val interpArguments = List(
       "-Yrepl-class-based",
       "-Yrepl-outdir",
-      s"${spark.sparkContext.getConf.get("spark.repl.class.outputDir")}")
+      s"${spark.sparkContext.getConf.get("spark.repl.class.outputDir")}",
+      "-classpath",
+      allJars.mkString(File.pathSeparator))
     settings.processArguments(interpArguments, processAll = true)
     settings.usejavacp.value = true
-    val currentClassLoader = Thread.currentThread().getContextClassLoader
     settings.embeddedDefaults(currentClassLoader)
     this.createInterpreter()
     this.initializeSynchronous()
     try {
       this.compilerClasspath
       this.ensureClassLoader()
-      var classLoader: ClassLoader = Thread.currentThread().getContextClassLoader
-      while (classLoader != null) {
-        classLoader match {
-          case loader: MutableURLClassLoader =>
-            val allJars = loader.getURLs.filter { u =>
-              val file = new File(u.getPath)
-              u.getProtocol == "file" && file.isFile &&
-              file.getName.contains("scala-lang_scala-reflect")

Review Comment:
   sounds good, will remove it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


Re: [PR] [KYUUBI #6223] Fix Scala interpreter can not access spark.jars issue [kyuubi]

Posted by "turboFei (via GitHub)" <gi...@apache.org>.
turboFei commented on code in PR #6235:
URL: https://github.com/apache/kyuubi/pull/6235#discussion_r1547105582


##########
externals/kyuubi-spark-sql-engine/src/main/scala-2.12/org/apache/kyuubi/engine/spark/repl/KyuubiSparkILoop.scala:
##########
@@ -97,6 +86,23 @@ private[spark] case class KyuubiSparkILoop private (
     }
   }
 
+  private def getAllJars(currentClassLoader: ClassLoader): Array[URL] = {
+    var classLoader: ClassLoader = currentClassLoader
+    var allJars = Array.empty[URL]
+    while (classLoader != null) {
+      classLoader match {
+        case loader: MutableURLClassLoader =>
+          allJars = loader.getURLs.filter { u =>
+            u.getProtocol == "file" && new File(u.getPath).isFile

Review Comment:
   will add comments after the UT finished.
   ```
               // TODO: handle SPARK-47475 since Spark 4.0.0 in the future
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


Re: [PR] Fix Scala interpreter can not access spark.jars issue [kyuubi]

Posted by "turboFei (via GitHub)" <gi...@apache.org>.
turboFei commented on code in PR #6235:
URL: https://github.com/apache/kyuubi/pull/6235#discussion_r1547086187


##########
externals/kyuubi-spark-sql-engine/src/main/scala-2.12/org/apache/kyuubi/engine/spark/repl/KyuubiSparkILoop.scala:
##########
@@ -28,47 +29,35 @@ import org.apache.spark.repl.SparkILoop
 import org.apache.spark.sql.{DataFrame, SparkSession}
 import org.apache.spark.util.MutableURLClassLoader
 
-import org.apache.kyuubi.Utils
+import org.apache.kyuubi.{Logging, Utils}
 
 private[spark] case class KyuubiSparkILoop private (
     spark: SparkSession,
     output: ByteArrayOutputStream)
-  extends SparkILoop(None, new PrintWriter(output)) {
+  extends SparkILoop(None, new PrintWriter(output)) with Logging {
   import KyuubiSparkILoop._
 
   val result = new DataFrameHolder(spark)
 
   private def initialize(): Unit = withLockRequired {
+    val currentClassLoader = Thread.currentThread().getContextClassLoader
+    val allJars = getAllJars(currentClassLoader)
+    info(s"Adding jars to Scala interpreter's class path: ${allJars.mkString(File.pathSeparator)}")
     settings = new Settings
     val interpArguments = List(
       "-Yrepl-class-based",
       "-Yrepl-outdir",
-      s"${spark.sparkContext.getConf.get("spark.repl.class.outputDir")}")
+      s"${spark.sparkContext.getConf.get("spark.repl.class.outputDir")}",
+      "-classpath",
+      allJars.mkString(File.pathSeparator))

Review Comment:
   Tested with yarn-cluster mode:
   ```
   24/04/01 20:12:36 INFO ExecuteScala: Processing b_stf's query[874507af-4dfe-4521-ad26-3caeaa154d28]: PENDING_STATE -> RUNNING_STATE, statement:
   import org.apache.iceberg._
   24/04/01 20:12:36 INFO OperationAuditLogger: operation=874507af-4dfe-4521-ad26-3caeaa154d28	opType=ExecuteScala	state=RUNNING	user=b_stf	session=973b28ec-52d8-454e-b2f1-45fc95957615
   24/04/01 20:12:36 INFO ExecuteScala:
              Spark application name: kyuubi_apollorno_CONNECTION_SPARK_SQL_b_stf_973b28ec-52d8-454e-b2f1-45fc95957615
                    application ID: application_1710906451321_1347246
                    application web UI: https://apollo-rno-rm-1.vip.hadoop.ebay.com:50030/proxy/application_1710906451321_1347246,https://apollo-rno-rm-2.vip.hadoop.ebay.com:50030/proxy/application_1710906451321_1347246
                    master: yarn
                    deploy mode: cluster
                    version: 3.1.1.1.3.0
              Start time: 2024-04-01T20:11:29.244
              User: b_stf
   24/04/01 20:12:36 INFO ExecuteScala: Processing b_stf's query[874507af-4dfe-4521-ad26-3caeaa154d28]: RUNNING_STATE -> FINISHED_STATE, time taken: 0.154 seconds
   24/04/01 20:12:36 INFO OperationAuditLogger: operation=874507af-4dfe-4521-ad26-3caeaa154d28	opType=ExecuteScala	state=FINISHED	user=b_stf	session=973b28ec-52d8-454e-b2f1-45fc95957615
   24/04/01 20:12:36 INFO ExecuteScala: statementId=874507af-4dfe-4521-ad26-3caeaa154d28, operationRunTime=0 ms, operationCpuTime=0 ms
   2024-04-01 20:12:36.439 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[874507af-4dfe-4521-ad26-3caeaa154d28] in FINISHED_STATE
   2024-04-01 20:12:36.439 INFO org.apache.kyuubi.operation.ExecuteStatement: Processing b_stf's query[874507af-4dfe-4521-ad26-3caeaa154d28]: RUNNING_STATE -> FINISHED_STATE, time taken: 0.197 seconds
   +------------------------------+
   |            output            |
   +------------------------------+
   | import org.apache.iceberg._  |
   +------------------------------+
   1 row selected (6.231 seconds)
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


Re: [PR] [KYUUBI #6223] Fix Scala interpreter can not access spark.jars issue [kyuubi]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #6235:
URL: https://github.com/apache/kyuubi/pull/6235#issuecomment-2031126011

   ## [Codecov](https://app.codecov.io/gh/apache/kyuubi/pull/6235?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   All modified and coverable lines are covered by tests :white_check_mark:
   > Project coverage is 58.41%. Comparing base [(`d0448f2`)](https://app.codecov.io/gh/apache/kyuubi/commit/d0448f25205f06950d755b5c49084a4ce3b5fd04?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`a8b3731`)](https://app.codecov.io/gh/apache/kyuubi/pull/6235?dropdown=coverage&src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   
   
   <details><summary>Additional details and impacted files</summary>
   
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #6235      +/-   ##
   ============================================
   - Coverage     58.44%   58.41%   -0.03%     
     Complexity       24       24              
   ============================================
     Files           651      651              
     Lines         39509    39509              
     Branches       5439     5439              
   ============================================
   - Hits          23091    23080      -11     
   - Misses        13949    13954       +5     
   - Partials       2469     2475       +6     
   ```
   
   
   
   </details>
   
   [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/kyuubi/pull/6235?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


Re: [PR] Fix Scala interpreter can not access spark.jars issue [kyuubi]

Posted by "turboFei (via GitHub)" <gi...@apache.org>.
turboFei commented on code in PR #6235:
URL: https://github.com/apache/kyuubi/pull/6235#discussion_r1547086469


##########
externals/kyuubi-spark-sql-engine/src/main/scala-2.12/org/apache/kyuubi/engine/spark/repl/KyuubiSparkILoop.scala:
##########
@@ -28,47 +29,35 @@ import org.apache.spark.repl.SparkILoop
 import org.apache.spark.sql.{DataFrame, SparkSession}
 import org.apache.spark.util.MutableURLClassLoader
 
-import org.apache.kyuubi.Utils
+import org.apache.kyuubi.{Logging, Utils}
 
 private[spark] case class KyuubiSparkILoop private (
     spark: SparkSession,
     output: ByteArrayOutputStream)
-  extends SparkILoop(None, new PrintWriter(output)) {
+  extends SparkILoop(None, new PrintWriter(output)) with Logging {
   import KyuubiSparkILoop._
 
   val result = new DataFrameHolder(spark)
 
   private def initialize(): Unit = withLockRequired {
+    val currentClassLoader = Thread.currentThread().getContextClassLoader
+    val allJars = getAllJars(currentClassLoader)
+    info(s"Adding jars to Scala interpreter's class path: ${allJars.mkString(File.pathSeparator)}")
     settings = new Settings
     val interpArguments = List(
       "-Yrepl-class-based",
       "-Yrepl-outdir",
-      s"${spark.sparkContext.getConf.get("spark.repl.class.outputDir")}")
+      s"${spark.sparkContext.getConf.get("spark.repl.class.outputDir")}",
+      "-classpath",
+      allJars.mkString(File.pathSeparator))

Review Comment:
   I tried `this.addUrlsToClassPath(allJars: _*)`, it does not work for my use case - yarn-cluster mode.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


Re: [PR] [KYUUBI #6223] Fix Scala interpreter can not access spark.jars issue [kyuubi]

Posted by "turboFei (via GitHub)" <gi...@apache.org>.
turboFei commented on code in PR #6235:
URL: https://github.com/apache/kyuubi/pull/6235#discussion_r1548089829


##########
externals/kyuubi-spark-sql-engine/src/main/scala-2.12/org/apache/kyuubi/engine/spark/repl/KyuubiSparkILoop.scala:
##########
@@ -28,47 +29,35 @@ import org.apache.spark.repl.SparkILoop
 import org.apache.spark.sql.{DataFrame, SparkSession}
 import org.apache.spark.util.MutableURLClassLoader
 
-import org.apache.kyuubi.Utils
+import org.apache.kyuubi.{Logging, Utils}
 
 private[spark] case class KyuubiSparkILoop private (
     spark: SparkSession,
     output: ByteArrayOutputStream)
-  extends SparkILoop(None, new PrintWriter(output)) {
+  extends SparkILoop(None, new PrintWriter(output)) with Logging {
   import KyuubiSparkILoop._
 
   val result = new DataFrameHolder(spark)
 
   private def initialize(): Unit = withLockRequired {
+    val currentClassLoader = Thread.currentThread().getContextClassLoader
+    val interpreterClasspath = getAllJars(currentClassLoader).mkString(File.pathSeparator)
+    info(s"Adding jars to Scala interpreter's class path: $interpreterClasspath")
     settings = new Settings
     val interpArguments = List(
       "-Yrepl-class-based",
       "-Yrepl-outdir",
-      s"${spark.sparkContext.getConf.get("spark.repl.class.outputDir")}")
+      s"${spark.sparkContext.getConf.get("spark.repl.class.outputDir")}",
+      "-classpath",
+      interpreterClasspath)
     settings.processArguments(interpArguments, processAll = true)
     settings.usejavacp.value = true
-    val currentClassLoader = Thread.currentThread().getContextClassLoader
     settings.embeddedDefaults(currentClassLoader)
     this.createInterpreter()
     this.initializeSynchronous()
     try {
       this.compilerClasspath
       this.ensureClassLoader()
-      var classLoader: ClassLoader = Thread.currentThread().getContextClassLoader
-      while (classLoader != null) {
-        classLoader match {
-          case loader: MutableURLClassLoader =>
-            val allJars = loader.getURLs.filter { u =>
-              val file = new File(u.getPath)
-              u.getProtocol == "file" && file.isFile &&
-              file.getName.contains("scala-lang_scala-reflect")
-            }
-            this.addUrlsToClassPath(allJars: _*)

Review Comment:
   not sure.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


Re: [PR] Fix Scala interpreter can not access spark.jars issue [kyuubi]

Posted by "turboFei (via GitHub)" <gi...@apache.org>.
turboFei commented on code in PR #6235:
URL: https://github.com/apache/kyuubi/pull/6235#discussion_r1547086187


##########
externals/kyuubi-spark-sql-engine/src/main/scala-2.12/org/apache/kyuubi/engine/spark/repl/KyuubiSparkILoop.scala:
##########
@@ -28,47 +29,35 @@ import org.apache.spark.repl.SparkILoop
 import org.apache.spark.sql.{DataFrame, SparkSession}
 import org.apache.spark.util.MutableURLClassLoader
 
-import org.apache.kyuubi.Utils
+import org.apache.kyuubi.{Logging, Utils}
 
 private[spark] case class KyuubiSparkILoop private (
     spark: SparkSession,
     output: ByteArrayOutputStream)
-  extends SparkILoop(None, new PrintWriter(output)) {
+  extends SparkILoop(None, new PrintWriter(output)) with Logging {
   import KyuubiSparkILoop._
 
   val result = new DataFrameHolder(spark)
 
   private def initialize(): Unit = withLockRequired {
+    val currentClassLoader = Thread.currentThread().getContextClassLoader
+    val allJars = getAllJars(currentClassLoader)
+    info(s"Adding jars to Scala interpreter's class path: ${allJars.mkString(File.pathSeparator)}")
     settings = new Settings
     val interpArguments = List(
       "-Yrepl-class-based",
       "-Yrepl-outdir",
-      s"${spark.sparkContext.getConf.get("spark.repl.class.outputDir")}")
+      s"${spark.sparkContext.getConf.get("spark.repl.class.outputDir")}",
+      "-classpath",
+      allJars.mkString(File.pathSeparator))

Review Comment:
   Tested with yarn-cluster mode:
   ```
   24/04/01 20:12:36 INFO ExecuteScala: Processing b_stf's query[874507af-4dfe-4521-ad26-3caeaa154d28]: PENDING_STATE -> RUNNING_STATE, statement:
   import org.apache.iceberg._
   24/04/01 20:12:36 INFO OperationAuditLogger: operation=874507af-4dfe-4521-ad26-3caeaa154d28	opType=ExecuteScala	state=RUNNING	user=b_stf	session=973b28ec-52d8-454e-b2f1-45fc95957615
   24/04/01 20:12:36 INFO ExecuteScala:
              Spark application name: kyuubi_apollorno_CONNECTION_SPARK_SQL_b_stf_973b28ec-52d8-454e-b2f1-45fc95957615
                    application ID: application_1710906451321_1347246
                    application web UI:  ......
                    master: yarn
                    deploy mode: cluster
                    version: 3.1.1.1.3.0
              Start time: 2024-04-01T20:11:29.244
              User: b_stf
   24/04/01 20:12:36 INFO ExecuteScala: Processing b_stf's query[874507af-4dfe-4521-ad26-3caeaa154d28]: RUNNING_STATE -> FINISHED_STATE, time taken: 0.154 seconds
   24/04/01 20:12:36 INFO OperationAuditLogger: operation=874507af-4dfe-4521-ad26-3caeaa154d28	opType=ExecuteScala	state=FINISHED	user=b_stf	session=973b28ec-52d8-454e-b2f1-45fc95957615
   24/04/01 20:12:36 INFO ExecuteScala: statementId=874507af-4dfe-4521-ad26-3caeaa154d28, operationRunTime=0 ms, operationCpuTime=0 ms
   2024-04-01 20:12:36.439 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[874507af-4dfe-4521-ad26-3caeaa154d28] in FINISHED_STATE
   2024-04-01 20:12:36.439 INFO org.apache.kyuubi.operation.ExecuteStatement: Processing b_stf's query[874507af-4dfe-4521-ad26-3caeaa154d28]: RUNNING_STATE -> FINISHED_STATE, time taken: 0.197 seconds
   +------------------------------+
   |            output            |
   +------------------------------+
   | import org.apache.iceberg._  |
   +------------------------------+
   1 row selected (6.231 seconds)
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org