You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/11/29 00:31:08 UTC
[GitHub] [spark] ocworld opened a new pull request, #38828: [SPARK-35084][CORE] Spark 3: supporting --packages in k8s cluster mode
ocworld opened a new pull request, #38828:
URL: https://github.com/apache/spark/pull/38828
### What changes were proposed in this pull request?
Supporting '--packages' in the k8s cluster mode
### Why are the changes needed?
In spark 3, '--packages' in the k8s cluster mode is not supported. I expected that managing dependencies by using packages like spark 2.
Spark 2.4.5
https://github.com/apache/spark/blob/v2.4.5/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
```scala
if (!isMesosCluster && !isStandAloneCluster) {
// Resolve maven dependencies if there are any and add classpath to jars. Add them to py-files
// too for packages that include Python code
val resolvedMavenCoordinates = DependencyUtils.resolveMavenDependencies(
args.packagesExclusions, args.packages, args.repositories, args.ivyRepoPath,
args.ivySettingsPath)
if (!StringUtils.isBlank(resolvedMavenCoordinates)) {
args.jars = mergeFileLists(args.jars, resolvedMavenCoordinates)
if (args.isPython || isInternal(args.primaryResource)) {
args.pyFiles = mergeFileLists(args.pyFiles, resolvedMavenCoordinates)
}
}
// install any R packages that may have been passed through --jars or --packages.
// Spark Packages may contain R source code inside the jar.
if (args.isR && !StringUtils.isBlank(args.jars)) {
RPackageUtils.checkAndBuildRPackage(args.jars, printStream, args.verbose)
}
}
```
Spark 3.0.2
https://github.com/apache/spark/blob/v3.0.2/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
```scala
if (!StringUtils.isBlank(resolvedMavenCoordinates)) {
// In K8s client mode, when in the driver, add resolved jars early as we might need
// them at the submit time for artifact downloading.
// For example we might use the dependencies for downloading
// files from a Hadoop Compatible fs eg. S3. In this case the user might pass:
// --packages com.amazonaws:aws-java-sdk:1.7.4:org.apache.hadoop:hadoop-aws:2.7.6
if (isKubernetesClusterModeDriver) {
val loader = getSubmitClassLoader(sparkConf)
for (jar <- resolvedMavenCoordinates.split(",")) {
addJarToClasspath(jar, loader)
}
} else if (isKubernetesCluster) {
// We need this in K8s cluster mode so that we can upload local deps
// via the k8s application, like in cluster mode driver
childClasspath ++= resolvedMavenCoordinates.split(",")
} else {
args.jars = mergeFileLists(args.jars, resolvedMavenCoordinates)
if (args.isPython || isInternal(args.primaryResource)) {
args.pyFiles = mergeFileLists(args.pyFiles, resolvedMavenCoordinates)
}
}
}
```
unlike spark2, in spark 3, jars are not added in any place.
### Does this PR introduce _any_ user-facing change?
Unlike spark 2, resolved jars are added not in cluster mode spark submit but in driver.
It's because in spark 3, the feature is added that is uploading jars with prefix "file://" to s3.
So, if resolved jars are added in spark submit, every jars from packages are uploading to s3! When I tested it, it is very bad experience to me.
### How was this patch tested?
In my k8s environment, i tested the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] attilapiros commented on pull request #38828: [SPARK-35084][CORE] Spark 3: supporting --packages in k8s cluster mode
Posted by GitBox <gi...@apache.org>.
attilapiros commented on PR #38828:
URL: https://github.com/apache/spark/pull/38828#issuecomment-1380652767
I have started to review this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] attilapiros commented on pull request #38828: [SPARK-35084][CORE] Fixing --packages in k8s client mode with the driver running inside a POD
Posted by GitBox <gi...@apache.org>.
attilapiros commented on PR #38828:
URL: https://github.com/apache/spark/pull/38828#issuecomment-1383022543
@ocworld Congratulation for your first contribution!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #38828: [SPARK-35084][CORE] Spark 3: supporting --packages in k8s cluster mode
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on PR #38828:
URL: https://github.com/apache/spark/pull/38828#issuecomment-1330964153
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] roczei commented on pull request #38828: [SPARK-35084][CORE] Spark 3: supporting --packages in k8s cluster mode
Posted by GitBox <gi...@apache.org>.
roczei commented on PR #38828:
URL: https://github.com/apache/spark/pull/38828#issuecomment-1332308308
Hi @ocworld,
Thanks a lot for this fix! I have tested it and it works for me as well.
Do you plan to add unit tests? Have you found a solution for this problem what you have mentioned in the previous pull request? https://github.com/apache/spark/pull/32397#issuecomment-838655508
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] holdenk commented on pull request #38828: [SPARK-35084][CORE] Spark 3: supporting --packages in k8s cluster mode
Posted by GitBox <gi...@apache.org>.
holdenk commented on PR #38828:
URL: https://github.com/apache/spark/pull/38828#issuecomment-1358445005
LGTM unless any other folks have concerns.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] attilapiros commented on pull request #38828: [SPARK-35084][CORE] Spark 3: supporting --packages in k8s client mode with the driver running inside a POD
Posted by GitBox <gi...@apache.org>.
attilapiros commented on PR #38828:
URL: https://github.com/apache/spark/pull/38828#issuecomment-1382020838
LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] attilapiros commented on pull request #38828: [SPARK-35084][CORE] Fixing --packages in k8s client mode with the driver running inside a POD
Posted by GitBox <gi...@apache.org>.
attilapiros commented on PR #38828:
URL: https://github.com/apache/spark/pull/38828#issuecomment-1383016182
merged into master, thank you @ocworld
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ocworld commented on a diff in pull request #38828: [SPARK-35084][CORE] Spark 3: supporting --packages in k8s client mode with the driver running inside a POD
Posted by GitBox <gi...@apache.org>.
ocworld commented on code in PR #38828:
URL: https://github.com/apache/spark/pull/38828#discussion_r1069306481
##########
core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala:
##########
@@ -319,16 +319,18 @@ private[spark] class SparkSubmit extends Logging {
// For example we might use the dependencies for downloading
// files from a Hadoop Compatible fs e.g. S3. In this case the user might pass:
// --packages com.amazonaws:aws-java-sdk:1.7.4:org.apache.hadoop:hadoop-aws:2.7.6
Review Comment:
it is moved now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ocworld commented on pull request #38828: [SPARK-35084][CORE] Fixing --packages in k8s client mode with the driver running inside a POD
Posted by GitBox <gi...@apache.org>.
ocworld commented on PR #38828:
URL: https://github.com/apache/spark/pull/38828#issuecomment-1383026476
> @ocworld Congratulation for your first contribution!
Thank you for your review, merge and congratulation 😀 It is very excited to me!
@roczei Thanks also to your testing and your help for several months
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] attilapiros commented on a diff in pull request #38828: [SPARK-35084][CORE] Spark 3: supporting --packages in k8s cluster mode
Posted by GitBox <gi...@apache.org>.
attilapiros commented on code in PR #38828:
URL: https://github.com/apache/spark/pull/38828#discussion_r1068765407
##########
core/src/test/scala/org/apache/spark/deploy/SparkSubmitSuite.scala:
##########
@@ -486,6 +486,34 @@ class SparkSubmitSuite
conf.get("spark.kubernetes.driver.container.image") should be ("bar")
}
+ test("SPARK-35084: includes jars passed in through --packages in k8s client driver mode") {
Review Comment:
For me client driver mode sounds out of the terminology.
What about something like
"SPARK-35084: include jars of the --packages in k8s client mode & driver runs inside a POD"?
##########
core/src/test/scala/org/apache/spark/deploy/SparkSubmitSuite.scala:
##########
@@ -486,6 +486,34 @@ class SparkSubmitSuite
conf.get("spark.kubernetes.driver.container.image") should be ("bar")
}
+ test("SPARK-35084: includes jars passed in through --packages in k8s client driver mode") {
+ val unusedJar = TestUtils.createJarWithClasses(Seq.empty)
+ val main = MavenCoordinate("my.great.lib", "mylib", "0.1")
+ val dep = MavenCoordinate("my.great.dep", "mylib", "0.1")
+ IvyTestUtils.withRepository(main, Some(dep.toString), None) { repo =>
+ val clArgs = Seq(
+ "--class", JarCreationTest.getClass.getName.stripSuffix("$"),
+ "--name", "testApp",
+ "--deploy-mode", "client",
Review Comment:
This is correct! It is client mode!
##########
core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala:
##########
@@ -319,16 +319,18 @@ private[spark] class SparkSubmit extends Logging {
// For example we might use the dependencies for downloading
// files from a Hadoop Compatible fs e.g. S3. In this case the user might pass:
// --packages com.amazonaws:aws-java-sdk:1.7.4:org.apache.hadoop:hadoop-aws:2.7.6
Review Comment:
This comment block starting with `In K8s client mode, when in the driver` belongs to the condition:
```
if (isKubernetesClusterModeDriver) {
```
So please move that comment block above the mentioned condition!
##########
core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala:
##########
@@ -319,16 +319,18 @@ private[spark] class SparkSubmit extends Logging {
// For example we might use the dependencies for downloading
// files from a Hadoop Compatible fs e.g. S3. In this case the user might pass:
// --packages com.amazonaws:aws-java-sdk:1.7.4:org.apache.hadoop:hadoop-aws:2.7.6
- if (isKubernetesClusterModeDriver) {
- val loader = getSubmitClassLoader(sparkConf)
- for (jar <- resolvedMavenCoordinates) {
- addJarToClasspath(jar, loader)
- }
- } else if (isKubernetesCluster) {
+ if (isKubernetesCluster) {
// We need this in K8s cluster mode so that we can upload local deps
// via the k8s application, like in cluster mode driver
childClasspath ++= resolvedMavenCoordinates
} else {
+ if (isKubernetesClusterModeDriver) {
Review Comment:
@ocworld this is not your mistake but this variable name seems to be wrong. As it is in client mode (see its initial value where the `isKubernetesClient` is referenced) and check the documentation: https://spark.apache.org/docs/latest/running-on-kubernetes.html#client-mode.
@skonto if you agree we can fix this in a separate PR to avoid further confusions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] roczei commented on pull request #38828: [SPARK-35084][CORE] Spark 3: supporting --packages in k8s cluster mode
Posted by GitBox <gi...@apache.org>.
roczei commented on PR #38828:
URL: https://github.com/apache/spark/pull/38828#issuecomment-1354933514
@holdenk, @HyukjinKwon, @dongjoon-hyun
Could you please take a look when you have some time?
This fixes a k8s --packages issue which is part of Spark 3 since 3.0.0. It would be nice to solve it. Here you can see the old branch-3.0 where the conditional codes of the "if" / "else if" / "else" are equal to the latest master version:
https://github.com/apache/spark/blob/branch-3.0/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala#L316-L328
vs.
https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala#L317-L330
These conditions were restructured a bit by @ocworld I have already added my test results above. The fix works.
This K8S PR is a follow-up PR for #32397. It has been closed by github-action because it hasn't been updated in a while and there was no unit test. The requested unit test has been added, now we need just someone from the Spark committer team who can review it again. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ocworld commented on pull request #38828: [SPARK-35084][CORE] Spark 3: supporting --packages in k8s cluster mode
Posted by GitBox <gi...@apache.org>.
ocworld commented on PR #38828:
URL: https://github.com/apache/spark/pull/38828#issuecomment-1334610174
@roczei thank you for your testing and reply. A unittest testcase about it is added now.
It is not easy for me to add integraed test about the pr like my mention In #32397. Because I do not have any idea about capturing container inside file system and process’s information.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ocworld commented on pull request #38828: [SPARK-35084][CORE] Fixing --packages in k8s client mode with the driver running inside a POD
Posted by GitBox <gi...@apache.org>.
ocworld commented on PR #38828:
URL: https://github.com/apache/spark/pull/38828#issuecomment-1383029138
@AhnLab-OSS @AhnLab-OSSG
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ocworld commented on a diff in pull request #38828: [SPARK-35084][CORE] Spark 3: supporting --packages in k8s client mode with the driver running inside a POD
Posted by GitBox <gi...@apache.org>.
ocworld commented on code in PR #38828:
URL: https://github.com/apache/spark/pull/38828#discussion_r1069308430
##########
core/src/test/scala/org/apache/spark/deploy/SparkSubmitSuite.scala:
##########
@@ -486,6 +486,34 @@ class SparkSubmitSuite
conf.get("spark.kubernetes.driver.container.image") should be ("bar")
}
+ test("SPARK-35084: includes jars passed in through --packages in k8s client driver mode") {
Review Comment:
It is fixed ty!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] attilapiros closed pull request #38828: [SPARK-35084][CORE] Fixing --packages in k8s client mode with the driver running inside a POD
Posted by GitBox <gi...@apache.org>.
attilapiros closed pull request #38828: [SPARK-35084][CORE] Fixing --packages in k8s client mode with the driver running inside a POD
URL: https://github.com/apache/spark/pull/38828
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] roczei commented on pull request #38828: [SPARK-35084][CORE] Spark 3: supporting --packages in k8s cluster mode
Posted by GitBox <gi...@apache.org>.
roczei commented on PR #38828:
URL: https://github.com/apache/spark/pull/38828#issuecomment-1373446915
Thanks @holdenk for the review! When do you plan to merge it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ocworld commented on pull request #38828: [SPARK-35084][CORE] Spark 3: supporting --packages in k8s client mode with the driver running inside a POD
Posted by GitBox <gi...@apache.org>.
ocworld commented on PR #38828:
URL: https://github.com/apache/spark/pull/38828#issuecomment-1381749296
> First of all thanks for the fix! There is some tiny things but the code is fine.
>
> Please also fix the PR title "cluster mode" => "client mode with the driver running inside a POD".
Thanks you for your review. Except variable name "isKubernetesClusterModeDriver", every your comments is applied.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ocworld commented on pull request #38828: [SPARK-35084][CORE] Spark 3: supporting --packages in k8s client mode with the driver running inside a POD
Posted by GitBox <gi...@apache.org>.
ocworld commented on PR #38828:
URL: https://github.com/apache/spark/pull/38828#issuecomment-1381740448
> First of all thanks for the fix! There is some tiny things but the code is fine.
>
> Please also fix the PR title "cluster mode" => "client mode with the driver running inside a POD".
this pr title is fixed now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] roczei commented on pull request #38828: [SPARK-35084][CORE] Spark 3: supporting --packages in k8s cluster mode
Posted by GitBox <gi...@apache.org>.
roczei commented on PR #38828:
URL: https://github.com/apache/spark/pull/38828#issuecomment-1339222364
Thanks @ocworld for the uploaded unit test! It works perfectly, it can identify the issue.
Good:
```
- SPARK-35084: includes jars passed in through --packages in k8s client driver mode
```
Bad:
```
- SPARK-35084: includes jars passed in through --packages in k8s client driver mode *** FAILED ***
false was not equal to true (SparkSubmitSuite.scala:513)
```
Just for documentation. Here is my test case what I have validated in my environment manually. I am using a spark-shell in a k8s Docker container.
It fails with the following error without your fix:
/tmp/spark.properties:
```
spark.kubernetes.submitInDriver=true
spark.kubernetes.authenticate.driver.serviceAccountName=spark
spark.kubernetes.namespace=default
spark.driver.blockManager.port=7079
spark.driver.port=7078
spark.blockManager.port=7079
spark.kubernetes.executor.label.name=executor
spark.kubernetes.driver.label.name=driver
spark.locality.wait=0
spark.executor.instances=1
spark.kubernetes.container.image=spark:spark-35084-upstream-no-fix
spark.master=k8s\://https\://kubernetes.default.svc.cluster.local\:443
spark.jars.packages=com.github.music-of-the-ainur\:almaren-framework_2.12\:0.9.4-3.2,com.github.music-of-the-ainur\:http-almaren_2.12\:1.2.4-3.2
spark.driver.host=172.17.0.4
spark.kubernetes.driver.pod.name=spark-submitter-spark-35084-upstream-no-fix-rxpjs
```
```
spark-shell --properties-file /tmp/spark.properties
...
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 3.4.0-SNAPSHOT
/_/
Using Scala version 2.12.17 (OpenJDK 64-Bit Server VM, Java 17.0.5)
Type in expressions to have them evaluated.
Type :help for more information.
scala> import com.github.music.of.the.ainur.almaren.builder.Core.Implicit
<console>:22: error: object music is not a member of package com.github
import com.github.music.of.the.ainur.almaren.builder.Core.Implicit
^
scala> import com.github.music.of.the.ainur.almaren.Almaren
<console>:22: error: object music is not a member of package com.github
import com.github.music.of.the.ainur.almaren.Almaren
^
scala> import org.apache.spark.sql.DataFrame
import org.apache.spark.sql.DataFrame
scala> val almaren = Almaren("App Name")
<console>:26: error: not found: value Almaren
val almaren = Almaren("App Name")
^
scala> $intp.isettings.maxPrintString = 0
$intp.isettings.maxPrintString: Int = 0
scala> spark.conf.get("spark.jars")
res0: String = ""
```
and this is the good one which includes your fix:
/tmp/spark.properties
```
spark.kubernetes.submitInDriver=true
spark.kubernetes.authenticate.driver.serviceAccountName=spark
spark.kubernetes.namespace=default
spark.driver.blockManager.port=7079
spark.driver.port=7078
spark.blockManager.port=7079
spark.kubernetes.executor.label.name=executor
spark.kubernetes.driver.label.name=driver
spark.locality.wait=0
spark.executor.instances=1
spark.kubernetes.container.image=spark:spark-35084-upstream-with-fix
spark.master=k8s\://https\://kubernetes.default.svc.cluster.local\:443
spark.jars.packages=com.github.music-of-the-ainur\:almaren-framework_2.12\:0.9.4-3.2,com.github.music-of-the-ainur\:http-almaren_2.12\:1.2.4-3.2
spark.driver.host=172.17.0.3
spark.kubernetes.driver.pod.name=spark-submitter-spark-35084-upstream-with-fix-whxzl
```
```
spark-shell --properties-file /tmp/spark.properties
...
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 3.4.0-SNAPSHOT
/_/
Using Scala version 2.12.17 (OpenJDK 64-Bit Server VM, Java 17.0.5)
Type in expressions to have them evaluated.
Type :help for more information.
scala> import com.github.music.of.the.ainur.almaren.builder.Core.Implicit
import com.github.music.of.the.ainur.almaren.builder.Core.Implicit
scala> import com.github.music.of.the.ainur.almaren.Almaren
import com.github.music.of.the.ainur.almaren.Almaren
scala> import org.apache.spark.sql.DataFrame
import org.apache.spark.sql.DataFrame
scala> val almaren = Almaren("App Name")
almaren: com.github.music.of.the.ainur.almaren.Almaren.type = com.github.music.of.the.ainur.almaren.Almaren$@4c2f971
scala> $intp.isettings.maxPrintString = 0
$intp.isettings.maxPrintString: Int = 0
scala> spark.conf.get("spark.jars")
res0: String = file:///home/sparkuser/.ivy2/jars/com.github.music-of-the-ainur_almaren-framework_2.12-0.9.4-3.2.jar,file:///home/sparkuser/.ivy2/jars/com.github.music-of-the-ainur_http-almaren_2.12-1.2.4-3.2.jar,file:///home/sparkuser/.ivy2/jars/com.typesafe.scala-logging_scala-logging_2.12-3.9.4.jar,file:///home/sparkuser/.ivy2/jars/org.apache.spark_spark-avro_2.12-3.2.1.jar,file:///home/sparkuser/.ivy2/jars/com.databricks_spark-xml_2.12-0.14.0.jar,file:///home/sparkuser/.ivy2/jars/com.github.music-of-the-ainur_quenya-dsl_2.12-1.2.0-3.2.jar,file:///home/sparkuser/.ivy2/jars/org.scala-lang_scala-reflect-2.12.13.jar,file:///home/sparkuser/.ivy2/jars/org.slf4j_slf4j-api-1.7.30.jar,file:///home/sparkuser/.ivy2/jars/org.tukaani_xz-1.8.jar,file:///home/sparkuser/.ivy2/jars/org.spark-project.spark_unused-1.0.0.jar,file:///home/sparkuser/.ivy2/jars/commons-io_commons-io-2.8.0.jar,file:///home/sparkuser/.ivy2/jars/org.glassfish.jaxb_txw2-2.3.4.jar,file:///home/sparkuser/.ivy2/jars/org.apa
che.ws.xmlschema_xmlschema-core-2.2.5.jar,file:///home/sparkuser/.ivy2/jars/com.lihaoyi_requests_2.12-0.7.0.jar,file:///home/sparkuser/.ivy2/jars/com.lihaoyi_geny_2.12-0.6.10.jar
scala>
```
@dongjoon-hyun, @HyukjinKwon, @holdenk Could you please review / approve this pull request? Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org