You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@kyuubi.apache.org by "camper42 (via GitHub)" <gi...@apache.org> on 2024/05/31 08:23:46 UTC
[I] [Bug] Can't use a HDFS dir for `kyuubi.operation.result.saveToFile.dir` [kyuubi]
camper42 opened a new issue, #6437:
URL: https://github.com/apache/kyuubi/issues/6437
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [X] I have searched in the [issues](https://github.com/apache/kyuubi/issues?q=is%3Aissue) and found no similar issues.
### Describe the bug
we use JuiceFS as our HDFS. And have set `fs.jfs.impl` & `fs.AbstractFileSystem.jfs.impl` properly.
we set `spark.sql.warehouse.dir="jfs://datalake/hive"`
without `kyuubi.operation.result.saveToFile.dir=jfs://datalake/tmp`, everything works fine.
with `kyuubi.operation.result.saveToFile.dir=jfs://datalake/tmp`, Spark engine failed with error `java.io.IOException: JuiceFS initialized failed for jfs:///`.
Our friends at JuiceFS have discovered that `Paths.get` in `getEngineResultSavePath` may cause the problem.
```scala
scala> import java.nio.file.Paths
import java.nio.file.Paths
scala> Paths.get("jfs://datalake/tmp", "test_engine_id").toString
res0: String = jfs:/datalake/tmp/test_engine_id
scala> Paths.get("hdfs://datalake/tmp", "test_engine_id").toString
res1: String = hdfs:/datalake/tmp/test_engine_id
```
Should this be fixed or saveToFile only support local dir ?
### Affects Version(s)
1.9.0
### Kyuubi Server Log Output
_No response_
### Kyuubi Engine Log Output
```logtalk
24/05/30 10:51:41 WARN FileSystem: Failed to initialize fileystem jfs:/datalake/tmp/spark-f47b20fb20d44845898adcaa3ae3cb68/8d6fc3b6-eee9-4a66-885f-8cbd8a5365e7: java.io.IOException: JuiceFS initialized failed for jfs:///
```
### Kyuubi Server Configurations
```yaml
kyuubi.operation.result.saveToFile.dir=jfs://datalake/tmp
```
### Kyuubi Engine Configurations
_No response_
### Additional context
_No response_
### Are you willing to submit PR?
- [X] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix.
- [ ] No. I cannot submit a PR at this time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] Can't use a HDFS dir for `kyuubi.operation.result.saveToFile.dir` [kyuubi]
Posted by "camper42 (via GitHub)" <gi...@apache.org>.
camper42 commented on issue #6437:
URL: https://github.com/apache/kyuubi/issues/6437#issuecomment-2141687841
actually, `SparkSQLEngine` and `SparkSQLSessionManager` use `org.apache.hadoop.fs.Path` after retrive path from `getEngineResultSavePath` and `getSessionResultSavePath`. 😅
https://github.com/apache/kyuubi/blob/master/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/SparkSQLEngine.scala#L95
https://github.com/apache/kyuubi/blob/master/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/session/SparkSQLSessionManager.scala#L191
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] Can't use a HDFS dir for `kyuubi.operation.result.saveToFile.dir` [kyuubi]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #6437:
URL: https://github.com/apache/kyuubi/issues/6437#issuecomment-2141472566
Hello @camper42,
Thanks for finding the time to report the issue!
We really appreciate the community's efforts to improve Apache Kyuubi.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] Can't use a HDFS dir for `kyuubi.operation.result.saveToFile.dir` [kyuubi]
Posted by "wForget (via GitHub)" <gi...@apache.org>.
wForget commented on issue #6437:
URL: https://github.com/apache/kyuubi/issues/6437#issuecomment-2141627591
Oh, I see, that means `saveToFile.dir` is a path without scheme, should we use `org.apache.hadoop.fs.Path` to handle the path?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] Can't use a HDFS dir for `kyuubi.operation.result.saveToFile.dir` [kyuubi]
Posted by "camper42 (via GitHub)" <gi...@apache.org>.
camper42 commented on issue #6437:
URL: https://github.com/apache/kyuubi/issues/6437#issuecomment-2144227195
yes, will send a PR this afternoon
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] Can't use a HDFS dir for `kyuubi.operation.result.saveToFile.dir` [kyuubi]
Posted by "wForget (via GitHub)" <gi...@apache.org>.
wForget commented on issue #6437:
URL: https://github.com/apache/kyuubi/issues/6437#issuecomment-2144150126
Thanks @camper42 , do you mind sending a PR to fix this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] Can't use a HDFS dir for `kyuubi.operation.result.saveToFile.dir` [kyuubi]
Posted by "camper42 (via GitHub)" <gi...@apache.org>.
camper42 commented on issue #6437:
URL: https://github.com/apache/kyuubi/issues/6437#issuecomment-2141697326
1. `kyuubi.operation.result.saveToFile.dir=jfs://datalake/tmp`
2. `getEngineResultSavePath`/`getSessionResultSavePath`/`getOperationResultSavePath` use `java.nio.file.Paths` and return a Path like `jfs:/datalake/tmp` (`jfs://` -> `jfs:/`)
3. `SparkSQLEngine.start` failed at `fs.mkdirs` https://github.com/apache/kyuubi/blob/master/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/SparkSQLEngine.scala#L97
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] Can't use a HDFS dir for `kyuubi.operation.result.saveToFile.dir` [kyuubi]
Posted by "camper42 (via GitHub)" <gi...@apache.org>.
camper42 commented on issue #6437:
URL: https://github.com/apache/kyuubi/issues/6437#issuecomment-2141576700
@wForget
```
24/05/31 17:14:53 ERROR SparkSQLEngine:
Spark application name: kyuubi_CONNECTION_SPARK_SQL_camper42_43ac361b-3028-416f-ad3a-3a2438f4b39b
application ID: spark-5e63469a46dd4360bfc1e02187dbdfe0
application tags:
application web UI: http://spark-73dd2d8fcdedc2e0-driver-svc.kyuubi.svc:4040
master: k8s://https://10.104.0.1:443
version: 3.5.1
driver: [cpu: 2, mem: 8g]
executor: [cpu: 4, mem: 10g, maxNum: 25]
Start time: Fri May 31 17:14:25 CST 2024
End time: Fri May 31 17:14:53 CST 2024
User: camper42 (shared mode: CONNECTION)
State: STOPPED
Diagnostic: Failed to start SparkSQLEngine: JuiceFS initialized failed for jfs:///
org.apache.kyuubi.KyuubiException: Failed to start SparkSQLEngine: JuiceFS initialized failed for jfs:///
at org.apache.kyuubi.engine.spark.SparkSQLEngine$.$anonfun$startEngine$1(SparkSQLEngine.scala:371)
at org.apache.kyuubi.engine.spark.SparkSQLEngine$.$anonfun$startEngine$1$adapted(SparkSQLEngine.scala:340)
at scala.Option.foreach(Option.scala:407)
at org.apache.kyuubi.engine.spark.SparkSQLEngine$.startEngine(SparkSQLEngine.scala:340)
at org.apache.kyuubi.engine.spark.SparkSQLEngine$.main(SparkSQLEngine.scala:405)
at org.apache.kyuubi.engine.spark.SparkSQLEngine.main(SparkSQLEngine.scala)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1029)
at org.apache.spark.deploy.SparkSubmit.$anonfun$submit$2(SparkSubmit.scala:169)
at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:62)
at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:61)
at java.base/java.security.AccessController.doPrivileged(Unknown Source)
at java.base/javax.security.auth.Subject.doAs(Unknown Source)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:61)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:169)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:217)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1120)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1129)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.io.IOException: JuiceFS initialized failed for jfs:///
at com.juicefs.JuiceFileSystemImpl.initialize(JuiceFileSystemImpl.java:544)
at org.apache.hadoop.fs.FilterFileSystem.initialize(FilterFileSystem.java:98)
at com.juicefs.JuiceFileSystem.initialize(JuiceFileSystem.java:241)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3469)
at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:174)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3574)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3521)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:540)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:365)
at org.apache.kyuubi.engine.spark.SparkSQLEngine.start(SparkSQLEngine.scala:96)
at org.apache.kyuubi.engine.spark.SparkSQLEngine$.$anonfun$startEngine$1(SparkSQLEngine.scala:358)
... 24 more```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] Can't use a HDFS dir for `kyuubi.operation.result.saveToFile.dir` [kyuubi]
Posted by "wForget (via GitHub)" <gi...@apache.org>.
wForget commented on issue #6437:
URL: https://github.com/apache/kyuubi/issues/6437#issuecomment-2141592405
It looks like `juicefs.name` is not configured correctly.
https://github.com/juicedata/juicefs/blob/4c168477ca4b03cd2261a4e24c034643dc1d5d36/sdk/java/src/main/java/io/juicefs/JuiceFileSystemImpl.java#L337
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] Can't use a HDFS dir for `kyuubi.operation.result.saveToFile.dir` [kyuubi]
Posted by "wForget (via GitHub)" <gi...@apache.org>.
wForget commented on issue #6437:
URL: https://github.com/apache/kyuubi/issues/6437#issuecomment-2141562053
> Spark engine failed with error java.io.IOException: JuiceFS initialized failed for jfs:///.
@camper42 Can you provide a full error stack?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] Can't use a HDFS dir for `kyuubi.operation.result.saveToFile.dir` [kyuubi]
Posted by "camper42 (via GitHub)" <gi...@apache.org>.
camper42 commented on issue #6437:
URL: https://github.com/apache/kyuubi/issues/6437#issuecomment-2141617576
@wForget as I mentioned above, `getEngineResultSavePath/getSessionResultSavePath/getOperationResultSavePath` in `SparkSQLSessionManager` misuse `java.nio.file.Paths` to process `OPERATION_RESULT_SAVE_TO_FILE_DIR`
https://github.com/apache/kyuubi/blob/master/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/session/SparkSQLSessionManager.scala#L214-L226
so, even if I configured our jfs correctly, `getEngineResultSavePath` return a wrong path.
```scala
scala> import java.nio.file.Paths
import java.nio.file.Paths
scala> Paths.get("jfs://datalake/tmp", "test_engine_id").toString
res0: String = jfs:/datalake/tmp/test_engine_id
scala> Paths.get("hdfs://datalake/tmp", "test_engine_id").toString
res1: String = hdfs:/datalake/tmp/test_engine_id
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] Can't use a HDFS dir for `kyuubi.operation.result.saveToFile.dir` [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 closed issue #6437: [Bug] Can't use a HDFS dir for `kyuubi.operation.result.saveToFile.dir`
URL: https://github.com/apache/kyuubi/issues/6437
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org