You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@kyuubi.apache.org by "pan3793 (via GitHub)" <gi...@apache.org> on 2024/12/30 06:25:39 UTC
[PR] Support rolling `spark.kubernetes.file.upload.path` [kyuubi]
pan3793 opened a new pull request, #6876:
URL: https://github.com/apache/kyuubi/pull/6876
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/master/contributing/code/index.html
2. If the PR is related to an issue in https://github.com/apache/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug, and what versions are affected.
-->
The vanilla Spark neither support rolling nor expiration mechanism for `spark.kubernetes.file.upload.path`, if you use file system that does not support TTL, e.g. HDFS, additional cleanup mechanisms are needed to prevent the files in this directory from growing indefinitely.
This PR proposes to let `spark.kubernetes.file.upload.path` support placeholders `{{YEAR}}`, `{{MONTH}}` and `{{DAY}}` and introduce a switch `kyuubi.kubernetes.spark.autoCreateFileUploadPath.enabled` to let Kyuubi server create the directory with 777 permission automatically before submitting Spark application.
Usage: for example, the user can configure the below configurations in `kyuubi-defaults.conf` to enable monthly rolling support of `spark.kubernetes.file.upload.path`
```
kyuubi.kubernetes.spark.autoCreateFileUploadPath.enabled=true
spark.kubernetes.file.upload.path=hdfs://hadoop-cluster/spark-upload-{{YEAR}}{{MONTH}}
```
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
New UTs are added.
### Was this patch authored or co-authored using generative AI tooling?
<!--
If a generative AI tooling has been used in the process of authoring this patch, please include
phrase 'Generated-by: ' followed by the name of the tool and its version.
If no, write 'No'.
Please refer to the [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html) for details.
-->
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] Support rolling `spark.kubernetes.file.upload.path` [kyuubi]
Posted by "turboFei (via GitHub)" <gi...@apache.org>.
turboFei commented on PR #6876:
URL: https://github.com/apache/kyuubi/pull/6876#issuecomment-2565994996
will check it today.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] Support rolling `spark.kubernetes.file.upload.path` [kyuubi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #6876:
URL: https://github.com/apache/kyuubi/pull/6876#issuecomment-2565142095
## [Codecov](https://app.codecov.io/gh/apache/kyuubi/pull/6876?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
Attention: Patch coverage is `0%` with `42 lines` in your changes missing coverage. Please review.
> Project coverage is 0.00%. Comparing base [(`e8cbff3`)](https://app.codecov.io/gh/apache/kyuubi/commit/e8cbff32d4857d108bc8737fca5e74fab7e9e3de?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`3eade8b`)](https://app.codecov.io/gh/apache/kyuubi/commit/3eade8bc4bca981c54465ada19e8d3a5144f6d4c?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
> Report is 4 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/kyuubi/pull/6876?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...ache/kyuubi/engine/spark/SparkProcessBuilder.scala](https://app.codecov.io/gh/apache/kyuubi/pull/6876?src=pr&el=tree&filepath=kyuubi-server%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fkyuubi%2Fengine%2Fspark%2FSparkProcessBuilder.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9lbmdpbmUvc3BhcmsvU3BhcmtQcm9jZXNzQnVpbGRlci5zY2FsYQ==) | 0.00% | [35 Missing :warning: ](https://app.codecov.io/gh/apache/kyuubi/pull/6876?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...in/scala/org/apache/kyuubi/config/KyuubiConf.scala](https://app.codecov.io/gh/apache/kyuubi/pull/6876?src=pr&el=tree&filepath=kyuubi-common%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fkyuubi%2Fconfig%2FKyuubiConf.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-a3l1dWJpLWNvbW1vbi9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9jb25maWcvS3l1dWJpQ29uZi5zY2FsYQ==) | 0.00% | [5 Missing :warning: ](https://app.codecov.io/gh/apache/kyuubi/pull/6876?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...kyuubi/engine/spark/SparkBatchProcessBuilder.scala](https://app.codecov.io/gh/apache/kyuubi/pull/6876?src=pr&el=tree&filepath=kyuubi-server%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fkyuubi%2Fengine%2Fspark%2FSparkBatchProcessBuilder.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9lbmdpbmUvc3BhcmsvU3BhcmtCYXRjaFByb2Nlc3NCdWlsZGVyLnNjYWxh) | 0.00% | [2 Missing :warning: ](https://app.codecov.io/gh/apache/kyuubi/pull/6876?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #6876 +/- ##
======================================
Coverage 0.00% 0.00%
======================================
Files 688 688
Lines 42545 42591 +46
Branches 5800 5806 +6
======================================
- Misses 42545 42591 +46
```
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/kyuubi/pull/6876?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] Support rolling `spark.kubernetes.file.upload.path` [kyuubi]
Posted by "zwangsheng (via GitHub)" <gi...@apache.org>.
zwangsheng commented on code in PR #6876:
URL: https://github.com/apache/kyuubi/pull/6876#discussion_r1899881820
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/engine/spark/SparkProcessBuilder.scala:
##########
@@ -294,6 +336,14 @@ class SparkProcessBuilder(
}
}
+ def isK8sClusterMode: Boolean = {
Review Comment:
clusterManager().exists(cm => cm.toLowerCase(Locale.ROOT).startsWith("k8s")) && deployMode().exists(_.toLowerCase(Locale.ROOT) == "cluster")
##########
docs/deployment/engine_on_kubernetes.md:
##########
@@ -48,6 +48,12 @@ The minimum required configurations are:
* spark.kubernetes.file.upload.path (path on S3 or HDFS)
* spark.kubernetes.authenticate.driver.serviceAccountName ([viz ServiceAccount](#serviceaccount))
+The vanilla Spark neither support rolling nor expiration mechanism for `spark.kubernetes.file.upload.path`, if you use
+file system that does not support TTL, e.g. HDFS, additional cleanup mechanisms are needed to prevent the files in this
+directory from growing indefinitely. Since Kyuubi v1.11.0, you can configure `spark.kubernetes.file.upload.path` with
+placeholders `{{YEAR}}`, `{{MONTH}}` and `{{DAY}}`, and enable `kyuubi.kubernetes.spark.autoCreateFileUploadPath.enabled`
+to let Kyuubi server create the directory with 777 permission automatically before submitting Spark application.
+
Review Comment:
It seems that our current implementation does not solve the problem of file growth. Will this issue be solved in subsequent PRs?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] Support rolling `spark.kubernetes.file.upload.path` [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #6876:
URL: https://github.com/apache/kyuubi/pull/6876#issuecomment-2566076304
> Spark would create sub dir s"spark-upload-${UUID.randomUUID()}" under the `spark.kubernetes.file.upload.path` for each uploading, the administer still need to cleanup the staging directory periodically.
Yes, exactly. Previously, it was unsafe to delete the whole `spark.kubernetes.file.upload.path` directly because the submitting Spark apps might use it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] Support rolling `spark.kubernetes.file.upload.path` [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #6876:
URL: https://github.com/apache/kyuubi/pull/6876#discussion_r1899883051
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/engine/spark/SparkProcessBuilder.scala:
##########
@@ -266,6 +274,40 @@ class SparkProcessBuilder(
map.result().toMap
}
+ def prepareK8sFileUploadPath(): Map[String, String] = {
+ kubernetesFileUploadPath() match {
+ case Some(uploadPathPattern) if isK8sClusterMode =>
+ val today = LocalDate.now()
+ val uploadPath = uploadPathPattern
+ .replace("{{YEAR}}", today.format(YEAR_FMT))
+ .replace("{{MONTH}}", today.format(MONTH_FMT))
+ .replace("{{DAY}}", today.format(DAY_FMT))
+
+ if (conf.get(KUBERNETES_SPARK_AUTO_CREATE_FILE_UPLOAD_PATH)) {
+ val hadoopConf = KyuubiHadoopUtils.newHadoopConf(conf, loadDefaults = false)
+ val path = new Path(uploadPath)
+ var fs: FileSystem = null
+ try {
+ fs = path.getFileSystem(hadoopConf)
+ if (!fs.exists(path)) {
+ info(s"Try creating $KUBERNETES_FILE_UPLOAD_PATH: $uploadPath")
+ fs.mkdirs(path, KUBERNETES_UPLOAD_PATH_PERMISSION)
+ }
+ } catch {
+ case ioe: IOException =>
+ warn(s"Failed to create $KUBERNETES_FILE_UPLOAD_PATH: $uploadPath", ioe)
+ } finally {
+ if (fs != null) {
+ Utils.tryLogNonFatalError(fs.close())
+ }
+ }
+ }
+ Map(KUBERNETES_FILE_UPLOAD_PATH -> uploadPath)
Review Comment:
The key difference is the permission of `uploadPath`, Spark creates the whole dir using default permission 511, it might prevent other users from creating `randomDirName` under `uploadPath` later.
```
fs.mkdirs(new Path(s"${uploadPath}/${randomDirName}"))
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] Support rolling `spark.kubernetes.file.upload.path` [kyuubi]
Posted by "turboFei (via GitHub)" <gi...@apache.org>.
turboFei commented on PR #6876:
URL: https://github.com/apache/kyuubi/pull/6876#issuecomment-2566045193
Note that:
```
kyuubi.kubernetes.spark.autoCreateFileUploadPath.enabled=true
spark.kubernetes.file.upload.path=hdfs://hadoop-cluster/spark-upload-{{YEAR}}{{MONTH}}
```
Spark would create sub dir `s"spark-upload-${UUID.randomUUID()}"` under the `spark.kubernetes.file.upload.path` for each uploading, the administer still need to cleanup the staging directory periodically.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] Support rolling `spark.kubernetes.file.upload.path` [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #6876:
URL: https://github.com/apache/kyuubi/pull/6876#issuecomment-2590638645
Thanks, merged to master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] Support rolling `spark.kubernetes.file.upload.path` [kyuubi]
Posted by "turboFei (via GitHub)" <gi...@apache.org>.
turboFei commented on code in PR #6876:
URL: https://github.com/apache/kyuubi/pull/6876#discussion_r1899886692
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/engine/spark/SparkProcessBuilder.scala:
##########
@@ -266,6 +274,40 @@ class SparkProcessBuilder(
map.result().toMap
}
+ def prepareK8sFileUploadPath(): Map[String, String] = {
+ kubernetesFileUploadPath() match {
+ case Some(uploadPathPattern) if isK8sClusterMode =>
+ val today = LocalDate.now()
+ val uploadPath = uploadPathPattern
+ .replace("{{YEAR}}", today.format(YEAR_FMT))
+ .replace("{{MONTH}}", today.format(MONTH_FMT))
+ .replace("{{DAY}}", today.format(DAY_FMT))
+
+ if (conf.get(KUBERNETES_SPARK_AUTO_CREATE_FILE_UPLOAD_PATH)) {
+ val hadoopConf = KyuubiHadoopUtils.newHadoopConf(conf, loadDefaults = false)
+ val path = new Path(uploadPath)
+ var fs: FileSystem = null
+ try {
+ fs = path.getFileSystem(hadoopConf)
+ if (!fs.exists(path)) {
+ info(s"Try creating $KUBERNETES_FILE_UPLOAD_PATH: $uploadPath")
+ fs.mkdirs(path, KUBERNETES_UPLOAD_PATH_PERMISSION)
+ }
+ } catch {
+ case ioe: IOException =>
+ warn(s"Failed to create $KUBERNETES_FILE_UPLOAD_PATH: $uploadPath", ioe)
+ } finally {
+ if (fs != null) {
+ Utils.tryLogNonFatalError(fs.close())
+ }
+ }
+ }
+ Map(KUBERNETES_FILE_UPLOAD_PATH -> uploadPath)
Review Comment:
got it, thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] Support rolling `spark.kubernetes.file.upload.path` [kyuubi]
Posted by "zwangsheng (via GitHub)" <gi...@apache.org>.
zwangsheng commented on code in PR #6876:
URL: https://github.com/apache/kyuubi/pull/6876#discussion_r1899881820
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/engine/spark/SparkProcessBuilder.scala:
##########
@@ -294,6 +336,14 @@ class SparkProcessBuilder(
}
}
+ def isK8sClusterMode: Boolean = {
Review Comment:
```scala
clusterManager().exists(cm => cm.toLowerCase(Locale.ROOT).startsWith("k8s")) && deployMode().exists(_.toLowerCase(Locale.ROOT) == "cluster")
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] Support rolling `spark.kubernetes.file.upload.path` [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #6876:
URL: https://github.com/apache/kyuubi/pull/6876#discussion_r1899887111
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/engine/spark/SparkProcessBuilder.scala:
##########
@@ -266,6 +274,40 @@ class SparkProcessBuilder(
map.result().toMap
}
+ def prepareK8sFileUploadPath(): Map[String, String] = {
+ kubernetesFileUploadPath() match {
+ case Some(uploadPathPattern) if isK8sClusterMode =>
+ val today = LocalDate.now()
+ val uploadPath = uploadPathPattern
+ .replace("{{YEAR}}", today.format(YEAR_FMT))
+ .replace("{{MONTH}}", today.format(MONTH_FMT))
+ .replace("{{DAY}}", today.format(DAY_FMT))
+
+ if (conf.get(KUBERNETES_SPARK_AUTO_CREATE_FILE_UPLOAD_PATH)) {
+ val hadoopConf = KyuubiHadoopUtils.newHadoopConf(conf, loadDefaults = false)
+ val path = new Path(uploadPath)
+ var fs: FileSystem = null
+ try {
+ fs = path.getFileSystem(hadoopConf)
+ if (!fs.exists(path)) {
+ info(s"Try creating $KUBERNETES_FILE_UPLOAD_PATH: $uploadPath")
+ fs.mkdirs(path, KUBERNETES_UPLOAD_PATH_PERMISSION)
+ }
+ } catch {
+ case ioe: IOException =>
+ warn(s"Failed to create $KUBERNETES_FILE_UPLOAD_PATH: $uploadPath", ioe)
+ } finally {
+ if (fs != null) {
+ Utils.tryLogNonFatalError(fs.close())
+ }
+ }
+ }
+ Map(KUBERNETES_FILE_UPLOAD_PATH -> uploadPath)
Review Comment:
let me also leave a comment on the code to mention that
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] Support rolling `spark.kubernetes.file.upload.path` [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #6876:
URL: https://github.com/apache/kyuubi/pull/6876#discussion_r1899884181
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/engine/spark/SparkProcessBuilder.scala:
##########
@@ -294,6 +336,14 @@ class SparkProcessBuilder(
}
}
+ def isK8sClusterMode: Boolean = {
Review Comment:
updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] Support rolling `spark.kubernetes.file.upload.path` [kyuubi]
Posted by "turboFei (via GitHub)" <gi...@apache.org>.
turboFei commented on code in PR #6876:
URL: https://github.com/apache/kyuubi/pull/6876#discussion_r1899868699
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/engine/spark/SparkProcessBuilder.scala:
##########
@@ -266,6 +274,40 @@ class SparkProcessBuilder(
map.result().toMap
}
+ def prepareK8sFileUploadPath(): Map[String, String] = {
+ kubernetesFileUploadPath() match {
+ case Some(uploadPathPattern) if isK8sClusterMode =>
+ val today = LocalDate.now()
+ val uploadPath = uploadPathPattern
+ .replace("{{YEAR}}", today.format(YEAR_FMT))
+ .replace("{{MONTH}}", today.format(MONTH_FMT))
+ .replace("{{DAY}}", today.format(DAY_FMT))
+
+ if (conf.get(KUBERNETES_SPARK_AUTO_CREATE_FILE_UPLOAD_PATH)) {
+ val hadoopConf = KyuubiHadoopUtils.newHadoopConf(conf, loadDefaults = false)
+ val path = new Path(uploadPath)
+ var fs: FileSystem = null
+ try {
+ fs = path.getFileSystem(hadoopConf)
+ if (!fs.exists(path)) {
+ info(s"Try creating $KUBERNETES_FILE_UPLOAD_PATH: $uploadPath")
+ fs.mkdirs(path, KUBERNETES_UPLOAD_PATH_PERMISSION)
+ }
+ } catch {
+ case ioe: IOException =>
+ warn(s"Failed to create $KUBERNETES_FILE_UPLOAD_PATH: $uploadPath", ioe)
+ } finally {
+ if (fs != null) {
+ Utils.tryLogNonFatalError(fs.close())
+ }
+ }
+ }
+ Map(KUBERNETES_FILE_UPLOAD_PATH -> uploadPath)
Review Comment:
so since this PR, it will replace the `uploadPathPattern` pattern if it is set and is k8s cluster mode anyway.
The only usage for `KUBERNETES_SPARK_AUTO_CREATE_FILE_UPLOAD_PATH` is to create the path.
Seems not necessary? I saw Spark will create the dir for `KUBERNETES_FILE_UPLOAD_PATH`.
https://github.com/apache/spark/blob/c4145db09ffeb0d0f9714a517ff9bacb018483e3/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala#L314-L321
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] Support rolling `spark.kubernetes.file.upload.path` [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 closed pull request #6876: Support rolling `spark.kubernetes.file.upload.path`
URL: https://github.com/apache/kyuubi/pull/6876
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] Support rolling `spark.kubernetes.file.upload.path` [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #6876:
URL: https://github.com/apache/kyuubi/pull/6876#discussion_r1899883051
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/engine/spark/SparkProcessBuilder.scala:
##########
@@ -266,6 +274,40 @@ class SparkProcessBuilder(
map.result().toMap
}
+ def prepareK8sFileUploadPath(): Map[String, String] = {
+ kubernetesFileUploadPath() match {
+ case Some(uploadPathPattern) if isK8sClusterMode =>
+ val today = LocalDate.now()
+ val uploadPath = uploadPathPattern
+ .replace("{{YEAR}}", today.format(YEAR_FMT))
+ .replace("{{MONTH}}", today.format(MONTH_FMT))
+ .replace("{{DAY}}", today.format(DAY_FMT))
+
+ if (conf.get(KUBERNETES_SPARK_AUTO_CREATE_FILE_UPLOAD_PATH)) {
+ val hadoopConf = KyuubiHadoopUtils.newHadoopConf(conf, loadDefaults = false)
+ val path = new Path(uploadPath)
+ var fs: FileSystem = null
+ try {
+ fs = path.getFileSystem(hadoopConf)
+ if (!fs.exists(path)) {
+ info(s"Try creating $KUBERNETES_FILE_UPLOAD_PATH: $uploadPath")
+ fs.mkdirs(path, KUBERNETES_UPLOAD_PATH_PERMISSION)
+ }
+ } catch {
+ case ioe: IOException =>
+ warn(s"Failed to create $KUBERNETES_FILE_UPLOAD_PATH: $uploadPath", ioe)
+ } finally {
+ if (fs != null) {
+ Utils.tryLogNonFatalError(fs.close())
+ }
+ }
+ }
+ Map(KUBERNETES_FILE_UPLOAD_PATH -> uploadPath)
Review Comment:
The key difference is the permission of `uploadPath`, Spark creates the whole dir using default permission 511, it might prevent other users creating `randomDirName` later.
```
fs.mkdirs(new Path(s"${uploadPath}/${randomDirName}"))
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] Support rolling `spark.kubernetes.file.upload.path` [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #6876:
URL: https://github.com/apache/kyuubi/pull/6876#discussion_r1899883051
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/engine/spark/SparkProcessBuilder.scala:
##########
@@ -266,6 +274,40 @@ class SparkProcessBuilder(
map.result().toMap
}
+ def prepareK8sFileUploadPath(): Map[String, String] = {
+ kubernetesFileUploadPath() match {
+ case Some(uploadPathPattern) if isK8sClusterMode =>
+ val today = LocalDate.now()
+ val uploadPath = uploadPathPattern
+ .replace("{{YEAR}}", today.format(YEAR_FMT))
+ .replace("{{MONTH}}", today.format(MONTH_FMT))
+ .replace("{{DAY}}", today.format(DAY_FMT))
+
+ if (conf.get(KUBERNETES_SPARK_AUTO_CREATE_FILE_UPLOAD_PATH)) {
+ val hadoopConf = KyuubiHadoopUtils.newHadoopConf(conf, loadDefaults = false)
+ val path = new Path(uploadPath)
+ var fs: FileSystem = null
+ try {
+ fs = path.getFileSystem(hadoopConf)
+ if (!fs.exists(path)) {
+ info(s"Try creating $KUBERNETES_FILE_UPLOAD_PATH: $uploadPath")
+ fs.mkdirs(path, KUBERNETES_UPLOAD_PATH_PERMISSION)
+ }
+ } catch {
+ case ioe: IOException =>
+ warn(s"Failed to create $KUBERNETES_FILE_UPLOAD_PATH: $uploadPath", ioe)
+ } finally {
+ if (fs != null) {
+ Utils.tryLogNonFatalError(fs.close())
+ }
+ }
+ }
+ Map(KUBERNETES_FILE_UPLOAD_PATH -> uploadPath)
Review Comment:
The key difference is the permission of `uploadPath`, Spark creates the whole dir using default permission 511, it might prevent other users from creating `randomDirName` later.
```
fs.mkdirs(new Path(s"${uploadPath}/${randomDirName}"))
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] Support rolling `spark.kubernetes.file.upload.path` [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #6876:
URL: https://github.com/apache/kyuubi/pull/6876#discussion_r1899883656
##########
docs/deployment/engine_on_kubernetes.md:
##########
@@ -48,6 +48,12 @@ The minimum required configurations are:
* spark.kubernetes.file.upload.path (path on S3 or HDFS)
* spark.kubernetes.authenticate.driver.serviceAccountName ([viz ServiceAccount](#serviceaccount))
+The vanilla Spark neither support rolling nor expiration mechanism for `spark.kubernetes.file.upload.path`, if you use
+file system that does not support TTL, e.g. HDFS, additional cleanup mechanisms are needed to prevent the files in this
+directory from growing indefinitely. Since Kyuubi v1.11.0, you can configure `spark.kubernetes.file.upload.path` with
+placeholders `{{YEAR}}`, `{{MONTH}}` and `{{DAY}}`, and enable `kyuubi.kubernetes.spark.autoCreateFileUploadPath.enabled`
+to let Kyuubi server create the directory with 777 permission automatically before submitting Spark application.
+
Review Comment:
It adds the rolling support for `spark.kubernetes.file.upload.path`, for example,
```
spark.kubernetes.file.upload.path=hdfs://hadoop-testing/spark-upload-{{YEAR}}{{MONTH}}
```
```
hdfs://hadoop-testing/spark-upload-202412
hdfs://hadoop-testing/spark-upload-202501
```
Admin can safely delete the `hdfs://hadoop-testing/spark-upload-202412` after 20250101
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org