You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@kyuubi.apache.org by "turboFei (via GitHub)" <gi...@apache.org> on 2024/04/01 20:09:13 UTC
[PR] Scala repl urls [kyuubi]
turboFei opened a new pull request, #6235:
URL: https://github.com/apache/kyuubi/pull/6235
# :mag: Description
## Issue References ๐
<!-- Append the issue number after #. If there is no issue for you to link create one or -->
<!-- If there are no issues to link, please provide details here. -->
This pull request fixes #
## Describe Your Solution ๐ง
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
## Types of changes :bookmark:
<!--- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan ๐งช
#### Behavior Without This Pull Request :coffin:
#### Behavior With This Pull Request :tada:
#### Related Unit Tests
---
# Checklist ๐
<!--- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!--- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #6223] Fix Scala interpreter can not access spark.jars issue [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #6235:
URL: https://github.com/apache/kyuubi/pull/6235#issuecomment-2031143415
I verified this patch with Spark(3.3) on K8s cluster mode.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] Fix Scala interpreter can not access spark.jars issue [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #6235:
URL: https://github.com/apache/kyuubi/pull/6235#issuecomment-2031010587
cc @cxzl25
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] Fix Scala interpreter can not access spark.jars issue [kyuubi]
Posted by "turboFei (via GitHub)" <gi...@apache.org>.
turboFei commented on code in PR #6235:
URL: https://github.com/apache/kyuubi/pull/6235#discussion_r1547086187
##########
externals/kyuubi-spark-sql-engine/src/main/scala-2.12/org/apache/kyuubi/engine/spark/repl/KyuubiSparkILoop.scala:
##########
@@ -28,47 +29,35 @@ import org.apache.spark.repl.SparkILoop
import org.apache.spark.sql.{DataFrame, SparkSession}
import org.apache.spark.util.MutableURLClassLoader
-import org.apache.kyuubi.Utils
+import org.apache.kyuubi.{Logging, Utils}
private[spark] case class KyuubiSparkILoop private (
spark: SparkSession,
output: ByteArrayOutputStream)
- extends SparkILoop(None, new PrintWriter(output)) {
+ extends SparkILoop(None, new PrintWriter(output)) with Logging {
import KyuubiSparkILoop._
val result = new DataFrameHolder(spark)
private def initialize(): Unit = withLockRequired {
+ val currentClassLoader = Thread.currentThread().getContextClassLoader
+ val allJars = getAllJars(currentClassLoader)
+ info(s"Adding jars to Scala interpreter's class path: ${allJars.mkString(File.pathSeparator)}")
settings = new Settings
val interpArguments = List(
"-Yrepl-class-based",
"-Yrepl-outdir",
- s"${spark.sparkContext.getConf.get("spark.repl.class.outputDir")}")
+ s"${spark.sparkContext.getConf.get("spark.repl.class.outputDir")}",
+ "-classpath",
+ allJars.mkString(File.pathSeparator))
Review Comment:
Tested with yarn-cluster mode:
```
24/04/01 20:12:30 INFO KyuubiSparkILoop: Adding jars to Scala interpreter's class path: file:/hadoop/7/yarn/local/usercache/b_stf/appcache/application_1710906451321_1347246/container_e4072_1710906451321_1347246_01_000002/__app__.jar:file:/hadoop/7/yarn/local/usercache/b_stf/appcache/application_1710906451321_1347246/container_e4072_1710906451321_1347246_01_000002/delta-core_2.12-1.0.2.0.3.0.jar:file:/hadoop/7/yarn/local/usercache/b_stf/appcache/application_1710906451321_1347246/container_e4072_1710906451321_1347246_01_000002/iceberg-spark-runtime-3.1_2.12-1.2.1.0.7.1.jar:file:/hadoop/7/yarn/local/usercache/b_stf/appcache/application_1710906451321_1347246/container_e4072_1710906451321_1347246_01_000002/simplark-etl-assembly-latest-testing.jar:file:/hadoop/7/yarn/local/usercache/b_stf/appcache/application_1710906451321_1347246/container_e4072_1710906451321_1347246_01_000002/zeta-serde-0.0.4-jar-with-dependencies.jar
```
```
24/04/01 20:12:36 INFO ExecuteScala: Processing b_stf's query[874507af-4dfe-4521-ad26-3caeaa154d28]: PENDING_STATE -> RUNNING_STATE, statement:
import org.apache.iceberg._
24/04/01 20:12:36 INFO OperationAuditLogger: operation=874507af-4dfe-4521-ad26-3caeaa154d28 opType=ExecuteScala state=RUNNING user=b_stf session=973b28ec-52d8-454e-b2f1-45fc95957615
24/04/01 20:12:36 INFO ExecuteScala:
Spark application name: kyuubi_apollorno_CONNECTION_SPARK_SQL_b_stf_973b28ec-52d8-454e-b2f1-45fc95957615
application ID: application_1710906451321_1347246
application web UI: ......
master: yarn
deploy mode: cluster
version: 3.1.1.1.3.0
Start time: 2024-04-01T20:11:29.244
User: b_stf
24/04/01 20:12:36 INFO ExecuteScala: Processing b_stf's query[874507af-4dfe-4521-ad26-3caeaa154d28]: RUNNING_STATE -> FINISHED_STATE, time taken: 0.154 seconds
24/04/01 20:12:36 INFO OperationAuditLogger: operation=874507af-4dfe-4521-ad26-3caeaa154d28 opType=ExecuteScala state=FINISHED user=b_stf session=973b28ec-52d8-454e-b2f1-45fc95957615
24/04/01 20:12:36 INFO ExecuteScala: statementId=874507af-4dfe-4521-ad26-3caeaa154d28, operationRunTime=0 ms, operationCpuTime=0 ms
2024-04-01 20:12:36.439 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[874507af-4dfe-4521-ad26-3caeaa154d28] in FINISHED_STATE
2024-04-01 20:12:36.439 INFO org.apache.kyuubi.operation.ExecuteStatement: Processing b_stf's query[874507af-4dfe-4521-ad26-3caeaa154d28]: RUNNING_STATE -> FINISHED_STATE, time taken: 0.197 seconds
+------------------------------+
| output |
+------------------------------+
| import org.apache.iceberg._ |
+------------------------------+
1 row selected (6.231 seconds)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #6223] Fix Scala interpreter can not access spark.jars issue [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #6235:
URL: https://github.com/apache/kyuubi/pull/6235#issuecomment-2034210003
Thanks, merged to master/1.9/1.8
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] Fix Scala interpreter can not access spark.jars issue [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #6235:
URL: https://github.com/apache/kyuubi/pull/6235#discussion_r1547091947
##########
externals/kyuubi-spark-sql-engine/src/main/scala-2.12/org/apache/kyuubi/engine/spark/repl/KyuubiSparkILoop.scala:
##########
@@ -28,47 +29,35 @@ import org.apache.spark.repl.SparkILoop
import org.apache.spark.sql.{DataFrame, SparkSession}
import org.apache.spark.util.MutableURLClassLoader
-import org.apache.kyuubi.Utils
+import org.apache.kyuubi.{Logging, Utils}
private[spark] case class KyuubiSparkILoop private (
spark: SparkSession,
output: ByteArrayOutputStream)
- extends SparkILoop(None, new PrintWriter(output)) {
+ extends SparkILoop(None, new PrintWriter(output)) with Logging {
import KyuubiSparkILoop._
val result = new DataFrameHolder(spark)
private def initialize(): Unit = withLockRequired {
+ val currentClassLoader = Thread.currentThread().getContextClassLoader
+ val allJars = getAllJars(currentClassLoader)
+ info(s"Adding jars to Scala interpreter's class path: ${allJars.mkString(File.pathSeparator)}")
settings = new Settings
val interpArguments = List(
"-Yrepl-class-based",
"-Yrepl-outdir",
- s"${spark.sparkContext.getConf.get("spark.repl.class.outputDir")}")
+ s"${spark.sparkContext.getConf.get("spark.repl.class.outputDir")}",
+ "-classpath",
+ allJars.mkString(File.pathSeparator))
settings.processArguments(interpArguments, processAll = true)
settings.usejavacp.value = true
- val currentClassLoader = Thread.currentThread().getContextClassLoader
settings.embeddedDefaults(currentClassLoader)
this.createInterpreter()
this.initializeSynchronous()
try {
this.compilerClasspath
this.ensureClassLoader()
- var classLoader: ClassLoader = Thread.currentThread().getContextClassLoader
- while (classLoader != null) {
- classLoader match {
- case loader: MutableURLClassLoader =>
- val allJars = loader.getURLs.filter { u =>
- val file = new File(u.getPath)
- u.getProtocol == "file" && file.isFile &&
- file.getName.contains("scala-lang_scala-reflect")
Review Comment:
this actually is a bug, and it should not be an issue when user.classpath.first is false (default)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #6223] Fix Scala interpreter can not access spark.jars issue [kyuubi]
Posted by "turboFei (via GitHub)" <gi...@apache.org>.
turboFei commented on code in PR #6235:
URL: https://github.com/apache/kyuubi/pull/6235#discussion_r1547105582
##########
externals/kyuubi-spark-sql-engine/src/main/scala-2.12/org/apache/kyuubi/engine/spark/repl/KyuubiSparkILoop.scala:
##########
@@ -97,6 +86,23 @@ private[spark] case class KyuubiSparkILoop private (
}
}
+ private def getAllJars(currentClassLoader: ClassLoader): Array[URL] = {
+ var classLoader: ClassLoader = currentClassLoader
+ var allJars = Array.empty[URL]
+ while (classLoader != null) {
+ classLoader match {
+ case loader: MutableURLClassLoader =>
+ allJars = loader.getURLs.filter { u =>
+ u.getProtocol == "file" && new File(u.getPath).isFile
Review Comment:
will add comments after the UT finished.
```
// TODO: handle SPARK-47475 since Spark 4.0.0 in the future
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #6223] Fix Scala interpreter can not access spark.jars issue [kyuubi]
Posted by "Yikf (via GitHub)" <gi...@apache.org>.
Yikf commented on code in PR #6235:
URL: https://github.com/apache/kyuubi/pull/6235#discussion_r1547390077
##########
externals/kyuubi-spark-sql-engine/src/main/scala-2.12/org/apache/kyuubi/engine/spark/repl/KyuubiSparkILoop.scala:
##########
@@ -28,47 +29,35 @@ import org.apache.spark.repl.SparkILoop
import org.apache.spark.sql.{DataFrame, SparkSession}
import org.apache.spark.util.MutableURLClassLoader
-import org.apache.kyuubi.Utils
+import org.apache.kyuubi.{Logging, Utils}
private[spark] case class KyuubiSparkILoop private (
spark: SparkSession,
output: ByteArrayOutputStream)
- extends SparkILoop(None, new PrintWriter(output)) {
+ extends SparkILoop(None, new PrintWriter(output)) with Logging {
import KyuubiSparkILoop._
val result = new DataFrameHolder(spark)
private def initialize(): Unit = withLockRequired {
+ val currentClassLoader = Thread.currentThread().getContextClassLoader
+ val interpreterClasspath = getAllJars(currentClassLoader).mkString(File.pathSeparator)
+ info(s"Adding jars to Scala interpreter's class path: $interpreterClasspath")
settings = new Settings
val interpArguments = List(
"-Yrepl-class-based",
"-Yrepl-outdir",
- s"${spark.sparkContext.getConf.get("spark.repl.class.outputDir")}")
+ s"${spark.sparkContext.getConf.get("spark.repl.class.outputDir")}",
+ "-classpath",
+ interpreterClasspath)
settings.processArguments(interpArguments, processAll = true)
settings.usejavacp.value = true
- val currentClassLoader = Thread.currentThread().getContextClassLoader
settings.embeddedDefaults(currentClassLoader)
this.createInterpreter()
this.initializeSynchronous()
try {
this.compilerClasspath
this.ensureClassLoader()
- var classLoader: ClassLoader = Thread.currentThread().getContextClassLoader
- while (classLoader != null) {
- classLoader match {
- case loader: MutableURLClassLoader =>
- val allJars = loader.getURLs.filter { u =>
- val file = new File(u.getPath)
- u.getProtocol == "file" && file.isFile &&
- file.getName.contains("scala-lang_scala-reflect")
- }
- this.addUrlsToClassPath(allJars: _*)
Review Comment:
Just a question, Is there a fundamental difference between `-classpath` and `addUrlsToClassPath `? Why can `-classpath` solve this problem in this way?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #6223] Fix Scala interpreter can not access spark.jars issue [kyuubi]
Posted by "turboFei (via GitHub)" <gi...@apache.org>.
turboFei commented on code in PR #6235:
URL: https://github.com/apache/kyuubi/pull/6235#discussion_r1547111904
##########
externals/kyuubi-spark-sql-engine/src/main/scala-2.12/org/apache/kyuubi/engine/spark/repl/KyuubiSparkILoop.scala:
##########
@@ -97,6 +86,23 @@ private[spark] case class KyuubiSparkILoop private (
}
}
+ private def getAllJars(currentClassLoader: ClassLoader): Array[URL] = {
+ var classLoader: ClassLoader = currentClassLoader
+ var allJars = Array.empty[URL]
+ while (classLoader != null) {
+ classLoader match {
+ case loader: MutableURLClassLoader =>
+ allJars = loader.getURLs.filter { u =>
+ u.getProtocol == "file" && new File(u.getPath).isFile
Review Comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #6223] Fix Scala interpreter can not access spark.jars issue [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #6235:
URL: https://github.com/apache/kyuubi/pull/6235#discussion_r1547098059
##########
externals/kyuubi-spark-sql-engine/src/main/scala-2.12/org/apache/kyuubi/engine/spark/repl/KyuubiSparkILoop.scala:
##########
@@ -97,6 +86,23 @@ private[spark] case class KyuubiSparkILoop private (
}
}
+ private def getAllJars(currentClassLoader: ClassLoader): Array[URL] = {
+ var classLoader: ClassLoader = currentClassLoader
+ var allJars = Array.empty[URL]
+ while (classLoader != null) {
+ classLoader match {
+ case loader: MutableURLClassLoader =>
+ allJars = loader.getURLs.filter { u =>
+ u.getProtocol == "file" && new File(u.getPath).isFile
Review Comment:
it should work well in most cases, but we may need to handle SPARK-47475 in the future
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] Fix Scala interpreter can not access spark.jars issue [kyuubi]
Posted by "turboFei (via GitHub)" <gi...@apache.org>.
turboFei commented on code in PR #6235:
URL: https://github.com/apache/kyuubi/pull/6235#discussion_r1547050139
##########
externals/kyuubi-spark-sql-engine/src/main/scala-2.12/org/apache/kyuubi/engine/spark/repl/KyuubiSparkILoop.scala:
##########
@@ -60,8 +60,11 @@ private[spark] case class KyuubiSparkILoop private (
val allJars = loader.getURLs.filter { u =>
val file = new File(u.getPath)
u.getProtocol == "file" && file.isFile &&
- file.getName.contains("scala-lang_scala-reflect")
+ // Some bad spark packages depend on the wrong version of scala-reflect. Blacklist it.
+ !file.getName.contains("scala-lang_scala-reflect")
}
+ info(s"Adding jars to Scala interpreter's class path: " +
+ allJars.mkString(File.pathSeparator))
Review Comment:
```
24/04/01 19:01:30 INFO KyuubiSparkILoop: Adding jars to Scala interpreter's class path: file:/Users/fwang12/todo/kyuubi/externals/kyuubi-spark-sql-engine/target/kyuubi-spark-sql-engine_2.12-1.10.0-SNAPSHOT.jar:file:/var/folders/pr/9x9rt3j974zfn7p4f5dpf8m80000gq/T/kyuubi-32215772-8004-43c1-a070-1cb0efd79bf0/test-function-02417951-6b76-4cf7-bb4a-5fc20372c034.jar
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #6223] Fix Scala interpreter can not access spark.jars issue [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 closed pull request #6235: [KYUUBI #6223] Fix Scala interpreter can not access spark.jars issue
URL: https://github.com/apache/kyuubi/pull/6235
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #6223] Fix Scala interpreter can not access spark.jars issue [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #6235:
URL: https://github.com/apache/kyuubi/pull/6235#discussion_r1548845718
##########
externals/kyuubi-spark-sql-engine/src/main/scala-2.12/org/apache/kyuubi/engine/spark/repl/KyuubiSparkILoop.scala:
##########
@@ -28,47 +29,35 @@ import org.apache.spark.repl.SparkILoop
import org.apache.spark.sql.{DataFrame, SparkSession}
import org.apache.spark.util.MutableURLClassLoader
-import org.apache.kyuubi.Utils
+import org.apache.kyuubi.{Logging, Utils}
private[spark] case class KyuubiSparkILoop private (
spark: SparkSession,
output: ByteArrayOutputStream)
- extends SparkILoop(None, new PrintWriter(output)) {
+ extends SparkILoop(None, new PrintWriter(output)) with Logging {
import KyuubiSparkILoop._
val result = new DataFrameHolder(spark)
private def initialize(): Unit = withLockRequired {
+ val currentClassLoader = Thread.currentThread().getContextClassLoader
+ val interpreterClasspath = getAllJars(currentClassLoader).mkString(File.pathSeparator)
+ info(s"Adding jars to Scala interpreter's class path: $interpreterClasspath")
settings = new Settings
val interpArguments = List(
"-Yrepl-class-based",
"-Yrepl-outdir",
- s"${spark.sparkContext.getConf.get("spark.repl.class.outputDir")}")
+ s"${spark.sparkContext.getConf.get("spark.repl.class.outputDir")}",
+ "-classpath",
+ interpreterClasspath)
settings.processArguments(interpArguments, processAll = true)
settings.usejavacp.value = true
- val currentClassLoader = Thread.currentThread().getContextClassLoader
settings.embeddedDefaults(currentClassLoader)
this.createInterpreter()
this.initializeSynchronous()
try {
this.compilerClasspath
this.ensureClassLoader()
- var classLoader: ClassLoader = Thread.currentThread().getContextClassLoader
- while (classLoader != null) {
- classLoader match {
- case loader: MutableURLClassLoader =>
- val allJars = loader.getURLs.filter { u =>
- val file = new File(u.getPath)
- u.getProtocol == "file" && file.isFile &&
- file.getName.contains("scala-lang_scala-reflect")
- }
- this.addUrlsToClassPath(allJars: _*)
Review Comment:
Not sure about the underlying implementation, I checked the Spark and Zeppelin code, both use `-classpath`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] Fix Scala interpreter can not access spark.jars issue [kyuubi]
Posted by "turboFei (via GitHub)" <gi...@apache.org>.
turboFei commented on code in PR #6235:
URL: https://github.com/apache/kyuubi/pull/6235#discussion_r1547092607
##########
externals/kyuubi-spark-sql-engine/src/main/scala-2.12/org/apache/kyuubi/engine/spark/repl/KyuubiSparkILoop.scala:
##########
@@ -28,47 +29,35 @@ import org.apache.spark.repl.SparkILoop
import org.apache.spark.sql.{DataFrame, SparkSession}
import org.apache.spark.util.MutableURLClassLoader
-import org.apache.kyuubi.Utils
+import org.apache.kyuubi.{Logging, Utils}
private[spark] case class KyuubiSparkILoop private (
spark: SparkSession,
output: ByteArrayOutputStream)
- extends SparkILoop(None, new PrintWriter(output)) {
+ extends SparkILoop(None, new PrintWriter(output)) with Logging {
import KyuubiSparkILoop._
val result = new DataFrameHolder(spark)
private def initialize(): Unit = withLockRequired {
+ val currentClassLoader = Thread.currentThread().getContextClassLoader
+ val allJars = getAllJars(currentClassLoader)
+ info(s"Adding jars to Scala interpreter's class path: ${allJars.mkString(File.pathSeparator)}")
settings = new Settings
val interpArguments = List(
"-Yrepl-class-based",
"-Yrepl-outdir",
- s"${spark.sparkContext.getConf.get("spark.repl.class.outputDir")}")
+ s"${spark.sparkContext.getConf.get("spark.repl.class.outputDir")}",
+ "-classpath",
+ allJars.mkString(File.pathSeparator))
settings.processArguments(interpArguments, processAll = true)
settings.usejavacp.value = true
- val currentClassLoader = Thread.currentThread().getContextClassLoader
settings.embeddedDefaults(currentClassLoader)
this.createInterpreter()
this.initializeSynchronous()
try {
this.compilerClasspath
this.ensureClassLoader()
- var classLoader: ClassLoader = Thread.currentThread().getContextClassLoader
- while (classLoader != null) {
- classLoader match {
- case loader: MutableURLClassLoader =>
- val allJars = loader.getURLs.filter { u =>
- val file = new File(u.getPath)
- u.getProtocol == "file" && file.isFile &&
- file.getName.contains("scala-lang_scala-reflect")
Review Comment:
sounds good, will remove it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #6223] Fix Scala interpreter can not access spark.jars issue [kyuubi]
Posted by "turboFei (via GitHub)" <gi...@apache.org>.
turboFei commented on code in PR #6235:
URL: https://github.com/apache/kyuubi/pull/6235#discussion_r1547105582
##########
externals/kyuubi-spark-sql-engine/src/main/scala-2.12/org/apache/kyuubi/engine/spark/repl/KyuubiSparkILoop.scala:
##########
@@ -97,6 +86,23 @@ private[spark] case class KyuubiSparkILoop private (
}
}
+ private def getAllJars(currentClassLoader: ClassLoader): Array[URL] = {
+ var classLoader: ClassLoader = currentClassLoader
+ var allJars = Array.empty[URL]
+ while (classLoader != null) {
+ classLoader match {
+ case loader: MutableURLClassLoader =>
+ allJars = loader.getURLs.filter { u =>
+ u.getProtocol == "file" && new File(u.getPath).isFile
Review Comment:
will add comments after the UT finished.
```
// TODO: handle SPARK-47475 since Spark 4.0.0 in the future
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] Fix Scala interpreter can not access spark.jars issue [kyuubi]
Posted by "turboFei (via GitHub)" <gi...@apache.org>.
turboFei commented on code in PR #6235:
URL: https://github.com/apache/kyuubi/pull/6235#discussion_r1547086187
##########
externals/kyuubi-spark-sql-engine/src/main/scala-2.12/org/apache/kyuubi/engine/spark/repl/KyuubiSparkILoop.scala:
##########
@@ -28,47 +29,35 @@ import org.apache.spark.repl.SparkILoop
import org.apache.spark.sql.{DataFrame, SparkSession}
import org.apache.spark.util.MutableURLClassLoader
-import org.apache.kyuubi.Utils
+import org.apache.kyuubi.{Logging, Utils}
private[spark] case class KyuubiSparkILoop private (
spark: SparkSession,
output: ByteArrayOutputStream)
- extends SparkILoop(None, new PrintWriter(output)) {
+ extends SparkILoop(None, new PrintWriter(output)) with Logging {
import KyuubiSparkILoop._
val result = new DataFrameHolder(spark)
private def initialize(): Unit = withLockRequired {
+ val currentClassLoader = Thread.currentThread().getContextClassLoader
+ val allJars = getAllJars(currentClassLoader)
+ info(s"Adding jars to Scala interpreter's class path: ${allJars.mkString(File.pathSeparator)}")
settings = new Settings
val interpArguments = List(
"-Yrepl-class-based",
"-Yrepl-outdir",
- s"${spark.sparkContext.getConf.get("spark.repl.class.outputDir")}")
+ s"${spark.sparkContext.getConf.get("spark.repl.class.outputDir")}",
+ "-classpath",
+ allJars.mkString(File.pathSeparator))
Review Comment:
Tested with yarn-cluster mode:
```
24/04/01 20:12:36 INFO ExecuteScala: Processing b_stf's query[874507af-4dfe-4521-ad26-3caeaa154d28]: PENDING_STATE -> RUNNING_STATE, statement:
import org.apache.iceberg._
24/04/01 20:12:36 INFO OperationAuditLogger: operation=874507af-4dfe-4521-ad26-3caeaa154d28 opType=ExecuteScala state=RUNNING user=b_stf session=973b28ec-52d8-454e-b2f1-45fc95957615
24/04/01 20:12:36 INFO ExecuteScala:
Spark application name: kyuubi_apollorno_CONNECTION_SPARK_SQL_b_stf_973b28ec-52d8-454e-b2f1-45fc95957615
application ID: application_1710906451321_1347246
application web UI: https://apollo-rno-rm-1.vip.hadoop.ebay.com:50030/proxy/application_1710906451321_1347246,https://apollo-rno-rm-2.vip.hadoop.ebay.com:50030/proxy/application_1710906451321_1347246
master: yarn
deploy mode: cluster
version: 3.1.1.1.3.0
Start time: 2024-04-01T20:11:29.244
User: b_stf
24/04/01 20:12:36 INFO ExecuteScala: Processing b_stf's query[874507af-4dfe-4521-ad26-3caeaa154d28]: RUNNING_STATE -> FINISHED_STATE, time taken: 0.154 seconds
24/04/01 20:12:36 INFO OperationAuditLogger: operation=874507af-4dfe-4521-ad26-3caeaa154d28 opType=ExecuteScala state=FINISHED user=b_stf session=973b28ec-52d8-454e-b2f1-45fc95957615
24/04/01 20:12:36 INFO ExecuteScala: statementId=874507af-4dfe-4521-ad26-3caeaa154d28, operationRunTime=0 ms, operationCpuTime=0 ms
2024-04-01 20:12:36.439 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[874507af-4dfe-4521-ad26-3caeaa154d28] in FINISHED_STATE
2024-04-01 20:12:36.439 INFO org.apache.kyuubi.operation.ExecuteStatement: Processing b_stf's query[874507af-4dfe-4521-ad26-3caeaa154d28]: RUNNING_STATE -> FINISHED_STATE, time taken: 0.197 seconds
+------------------------------+
| output |
+------------------------------+
| import org.apache.iceberg._ |
+------------------------------+
1 row selected (6.231 seconds)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #6223] Fix Scala interpreter can not access spark.jars issue [kyuubi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #6235:
URL: https://github.com/apache/kyuubi/pull/6235#issuecomment-2031126011
## [Codecov](https://app.codecov.io/gh/apache/kyuubi/pull/6235?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Project coverage is 58.41%. Comparing base [(`d0448f2`)](https://app.codecov.io/gh/apache/kyuubi/commit/d0448f25205f06950d755b5c49084a4ce3b5fd04?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`a8b3731`)](https://app.codecov.io/gh/apache/kyuubi/pull/6235?dropdown=coverage&src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #6235 +/- ##
============================================
- Coverage 58.44% 58.41% -0.03%
Complexity 24 24
============================================
Files 651 651
Lines 39509 39509
Branches 5439 5439
============================================
- Hits 23091 23080 -11
- Misses 13949 13954 +5
- Partials 2469 2475 +6
```
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/kyuubi/pull/6235?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] Fix Scala interpreter can not access spark.jars issue [kyuubi]
Posted by "turboFei (via GitHub)" <gi...@apache.org>.
turboFei commented on code in PR #6235:
URL: https://github.com/apache/kyuubi/pull/6235#discussion_r1547086469
##########
externals/kyuubi-spark-sql-engine/src/main/scala-2.12/org/apache/kyuubi/engine/spark/repl/KyuubiSparkILoop.scala:
##########
@@ -28,47 +29,35 @@ import org.apache.spark.repl.SparkILoop
import org.apache.spark.sql.{DataFrame, SparkSession}
import org.apache.spark.util.MutableURLClassLoader
-import org.apache.kyuubi.Utils
+import org.apache.kyuubi.{Logging, Utils}
private[spark] case class KyuubiSparkILoop private (
spark: SparkSession,
output: ByteArrayOutputStream)
- extends SparkILoop(None, new PrintWriter(output)) {
+ extends SparkILoop(None, new PrintWriter(output)) with Logging {
import KyuubiSparkILoop._
val result = new DataFrameHolder(spark)
private def initialize(): Unit = withLockRequired {
+ val currentClassLoader = Thread.currentThread().getContextClassLoader
+ val allJars = getAllJars(currentClassLoader)
+ info(s"Adding jars to Scala interpreter's class path: ${allJars.mkString(File.pathSeparator)}")
settings = new Settings
val interpArguments = List(
"-Yrepl-class-based",
"-Yrepl-outdir",
- s"${spark.sparkContext.getConf.get("spark.repl.class.outputDir")}")
+ s"${spark.sparkContext.getConf.get("spark.repl.class.outputDir")}",
+ "-classpath",
+ allJars.mkString(File.pathSeparator))
Review Comment:
I tried `this.addUrlsToClassPath(allJars: _*)`, it does not work for my use case - yarn-cluster mode.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #6223] Fix Scala interpreter can not access spark.jars issue [kyuubi]
Posted by "turboFei (via GitHub)" <gi...@apache.org>.
turboFei commented on code in PR #6235:
URL: https://github.com/apache/kyuubi/pull/6235#discussion_r1548089829
##########
externals/kyuubi-spark-sql-engine/src/main/scala-2.12/org/apache/kyuubi/engine/spark/repl/KyuubiSparkILoop.scala:
##########
@@ -28,47 +29,35 @@ import org.apache.spark.repl.SparkILoop
import org.apache.spark.sql.{DataFrame, SparkSession}
import org.apache.spark.util.MutableURLClassLoader
-import org.apache.kyuubi.Utils
+import org.apache.kyuubi.{Logging, Utils}
private[spark] case class KyuubiSparkILoop private (
spark: SparkSession,
output: ByteArrayOutputStream)
- extends SparkILoop(None, new PrintWriter(output)) {
+ extends SparkILoop(None, new PrintWriter(output)) with Logging {
import KyuubiSparkILoop._
val result = new DataFrameHolder(spark)
private def initialize(): Unit = withLockRequired {
+ val currentClassLoader = Thread.currentThread().getContextClassLoader
+ val interpreterClasspath = getAllJars(currentClassLoader).mkString(File.pathSeparator)
+ info(s"Adding jars to Scala interpreter's class path: $interpreterClasspath")
settings = new Settings
val interpArguments = List(
"-Yrepl-class-based",
"-Yrepl-outdir",
- s"${spark.sparkContext.getConf.get("spark.repl.class.outputDir")}")
+ s"${spark.sparkContext.getConf.get("spark.repl.class.outputDir")}",
+ "-classpath",
+ interpreterClasspath)
settings.processArguments(interpArguments, processAll = true)
settings.usejavacp.value = true
- val currentClassLoader = Thread.currentThread().getContextClassLoader
settings.embeddedDefaults(currentClassLoader)
this.createInterpreter()
this.initializeSynchronous()
try {
this.compilerClasspath
this.ensureClassLoader()
- var classLoader: ClassLoader = Thread.currentThread().getContextClassLoader
- while (classLoader != null) {
- classLoader match {
- case loader: MutableURLClassLoader =>
- val allJars = loader.getURLs.filter { u =>
- val file = new File(u.getPath)
- u.getProtocol == "file" && file.isFile &&
- file.getName.contains("scala-lang_scala-reflect")
- }
- this.addUrlsToClassPath(allJars: _*)
Review Comment:
not sure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] Fix Scala interpreter can not access spark.jars issue [kyuubi]
Posted by "turboFei (via GitHub)" <gi...@apache.org>.
turboFei commented on code in PR #6235:
URL: https://github.com/apache/kyuubi/pull/6235#discussion_r1547086187
##########
externals/kyuubi-spark-sql-engine/src/main/scala-2.12/org/apache/kyuubi/engine/spark/repl/KyuubiSparkILoop.scala:
##########
@@ -28,47 +29,35 @@ import org.apache.spark.repl.SparkILoop
import org.apache.spark.sql.{DataFrame, SparkSession}
import org.apache.spark.util.MutableURLClassLoader
-import org.apache.kyuubi.Utils
+import org.apache.kyuubi.{Logging, Utils}
private[spark] case class KyuubiSparkILoop private (
spark: SparkSession,
output: ByteArrayOutputStream)
- extends SparkILoop(None, new PrintWriter(output)) {
+ extends SparkILoop(None, new PrintWriter(output)) with Logging {
import KyuubiSparkILoop._
val result = new DataFrameHolder(spark)
private def initialize(): Unit = withLockRequired {
+ val currentClassLoader = Thread.currentThread().getContextClassLoader
+ val allJars = getAllJars(currentClassLoader)
+ info(s"Adding jars to Scala interpreter's class path: ${allJars.mkString(File.pathSeparator)}")
settings = new Settings
val interpArguments = List(
"-Yrepl-class-based",
"-Yrepl-outdir",
- s"${spark.sparkContext.getConf.get("spark.repl.class.outputDir")}")
+ s"${spark.sparkContext.getConf.get("spark.repl.class.outputDir")}",
+ "-classpath",
+ allJars.mkString(File.pathSeparator))
Review Comment:
Tested with yarn-cluster mode:
```
24/04/01 20:12:36 INFO ExecuteScala: Processing b_stf's query[874507af-4dfe-4521-ad26-3caeaa154d28]: PENDING_STATE -> RUNNING_STATE, statement:
import org.apache.iceberg._
24/04/01 20:12:36 INFO OperationAuditLogger: operation=874507af-4dfe-4521-ad26-3caeaa154d28 opType=ExecuteScala state=RUNNING user=b_stf session=973b28ec-52d8-454e-b2f1-45fc95957615
24/04/01 20:12:36 INFO ExecuteScala:
Spark application name: kyuubi_apollorno_CONNECTION_SPARK_SQL_b_stf_973b28ec-52d8-454e-b2f1-45fc95957615
application ID: application_1710906451321_1347246
application web UI: ......
master: yarn
deploy mode: cluster
version: 3.1.1.1.3.0
Start time: 2024-04-01T20:11:29.244
User: b_stf
24/04/01 20:12:36 INFO ExecuteScala: Processing b_stf's query[874507af-4dfe-4521-ad26-3caeaa154d28]: RUNNING_STATE -> FINISHED_STATE, time taken: 0.154 seconds
24/04/01 20:12:36 INFO OperationAuditLogger: operation=874507af-4dfe-4521-ad26-3caeaa154d28 opType=ExecuteScala state=FINISHED user=b_stf session=973b28ec-52d8-454e-b2f1-45fc95957615
24/04/01 20:12:36 INFO ExecuteScala: statementId=874507af-4dfe-4521-ad26-3caeaa154d28, operationRunTime=0 ms, operationCpuTime=0 ms
2024-04-01 20:12:36.439 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[874507af-4dfe-4521-ad26-3caeaa154d28] in FINISHED_STATE
2024-04-01 20:12:36.439 INFO org.apache.kyuubi.operation.ExecuteStatement: Processing b_stf's query[874507af-4dfe-4521-ad26-3caeaa154d28]: RUNNING_STATE -> FINISHED_STATE, time taken: 0.197 seconds
+------------------------------+
| output |
+------------------------------+
| import org.apache.iceberg._ |
+------------------------------+
1 row selected (6.231 seconds)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org