You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@kyuubi.apache.org by "nqvuong1998 (via GitHub)" <gi...@apache.org> on 2024/09/26 04:09:28 UTC
[I] [FEATURE] Support SparkSQL on Standalone cluster [kyuubi]
nqvuong1998 opened a new issue, #6716:
URL: https://github.com/apache/kyuubi/issues/6716
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [X] I have searched in the [issues](https://github.com/apache/kyuubi/issues?q=is%3Aissue) and found no similar issues.
### Describe the feature
Kyuubi currently provides an excellent interface for running Spark SQL on distributed Spark environments. However, it lacks official support for running SparkSQL on a standalone Spark cluster. This feature request proposes adding support for SparkSQL execution on Spark Standalone clusters, allowing users to deploy Kyuubi with more flexibility in Spark cluster configurations.
### Motivation
_No response_
### Describe the solution
_No response_
### Additional context
_No response_
### Are you willing to submit PR?
- [ ] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to improve.
- [X] No. I cannot submit a PR at this time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [FEATURE] Support SparkSQL on Standalone cluster [kyuubi]
Posted by "medina325 (via GitHub)" <gi...@apache.org>.
medina325 commented on issue #6716:
URL: https://github.com/apache/kyuubi/issues/6716#issuecomment-3036167542
Yes, it definitely seems like it, I just assumed the official Kyuubi image would have every necessary jar installed by default. Sorry if it's a silly question, but the jars would be missing in my Kyuubi's environment, not my Spark environment, right? I configured Kyuubi to submit jobs in client mode.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [FEATURE] Support SparkSQL on Standalone cluster [kyuubi]
Posted by "yaooqinn (via GitHub)" <gi...@apache.org>.
yaooqinn commented on issue #6716:
URL: https://github.com/apache/kyuubi/issues/6716#issuecomment-2376147444
AFAIK, ApplicationOperation is irrelevant to submit engines
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [FEATURE] Support SparkSQL on Standalone cluster [kyuubi]
Posted by "yaooqinn (via GitHub)" <gi...@apache.org>.
yaooqinn commented on issue #6716:
URL: https://github.com/apache/kyuubi/issues/6716#issuecomment-3043332035
Yes, Kyuubi has to know at least SPARK_HOME to submit Spark apps. Unfortunately, the official Kyuubi image is not my area, I'm not sure whether it's bundled with Spark releases, and set with SPARK_HOME correctly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [FEATURE] Support SparkSQL on Standalone cluster [kyuubi]
Posted by "medina325 (via GitHub)" <gi...@apache.org>.
medina325 commented on issue #6716:
URL: https://github.com/apache/kyuubi/issues/6716#issuecomment-3035239759
That's good to know. However, I just can't seem to configure Apache Kyuubi properly to work with my standalone Spark cluster.
I can't, for the life of me, figure out how to properly set the env vars like:
- JAVA_HOME
- SPARK_HOME
- SPARK_ENGINE_HOME
- HIVE_HOME
- HADOOP_CONF_DIR
Should these paths refer to Kyuubi's container filesystem or the Spark worker’s?
I’m using:
- bitnami/spark:3.5 for Spark
- apache/kyuubi:1.9.4-all for Kyuubi
If I use the built-in spark binaries in the Kyuubi's image, then I am able to establish a connection, but if I set `spark.master` to point to my Spark cluster's master node, I start getting all kinds of problems.
Right now I'm stuck at `...org.apache.kyuubi.KyuubiSQLException: org.apache.kyuubi.KyuubiSQLException: Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/hive/conf/HiveConf`.
Any help would be really appreciated!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [FEATURE] Support SparkSQL on Standalone cluster [kyuubi]
Posted by "medina325 (via GitHub)" <gi...@apache.org>.
medina325 commented on issue #6716:
URL: https://github.com/apache/kyuubi/issues/6716#issuecomment-3044852526
Do you know where I could find help? I thought this repository was the right place...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [FEATURE] Support SparkSQL on Standalone cluster [kyuubi]
Posted by "yaooqinn (via GitHub)" <gi...@apache.org>.
yaooqinn commented on issue #6716:
URL: https://github.com/apache/kyuubi/issues/6716#issuecomment-2375896531
Technically, Standalone clusters are supported. What was the issue you actually encountered?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [FEATURE] Support SparkSQL on Standalone cluster [kyuubi]
Posted by "medina325 (via GitHub)" <gi...@apache.org>.
medina325 commented on issue #6716:
URL: https://github.com/apache/kyuubi/issues/6716#issuecomment-3035244417
@yaooqinn Would starting a discussion be the right way to ask for help, or is it okay to post it here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [FEATURE] Support SparkSQL on Standalone cluster [kyuubi]
Posted by "yaooqinn (via GitHub)" <gi...@apache.org>.
yaooqinn commented on issue #6716:
URL: https://github.com/apache/kyuubi/issues/6716#issuecomment-3036121957
looks like missing hive jars？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [FEATURE] Support SparkSQL on Standalone cluster [kyuubi]
Posted by "nqvuong1998 (via GitHub)" <gi...@apache.org>.
nqvuong1998 commented on issue #6716:
URL: https://github.com/apache/kyuubi/issues/6716#issuecomment-2375978585
Hi @yaooqinn , Apache Spark Kubernetes Operator supports deploying a [Spark standalone cluster on Kubernetes](https://github.com/apache/spark-kubernetes-operator/blob/main/examples/prod-cluster-with-three-workers.yaml). However, I have come across an issue where Kyuubi has not yet implemented a [Spark Standalone ApplicationOperation](https://github.com/apache/kyuubi/issues/6539#issuecomment-2231536423). Therefore, I created this feature request.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [FEATURE] Support SparkSQL on Standalone cluster [kyuubi]
Posted by "nqvuong1998 (via GitHub)" <gi...@apache.org>.
nqvuong1998 commented on issue #6716:
URL: https://github.com/apache/kyuubi/issues/6716#issuecomment-2375830478
cc @yaooqinn @pan3793
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org