You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@kyuubi.apache.org by GitBox <gi...@apache.org> on 2022/09/05 10:17:29 UTC
[GitHub] [incubator-kyuubi] kevinclcn opened a new issue, #3409: [FEATURE] Support Kyuubi Engine run on Alicloud MaxCompute and AWS Glue
kevinclcn opened a new issue, #3409:
URL: https://github.com/apache/incubator-kyuubi/issues/3409
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [X] I have searched in the [issues](https://github.com/apache/incubator-kyuubi/issues?q=is%3Aissue) and found no similar issues.
### Describe the feature
目前Kyuubi Engine可以运行在Yarn或K8s上以执行通过JDBC提交的任务，但在云原生环境里，通常云提供商都提供弹性的云计算资源，比如阿里云的MaxCompute和AWS Glue。如果Kyuubi Engine支持运行在MaxCompute和Glue上，可以大大降低Spark的运行成本和维护成本。
阿里云的通过MaxCompute运行spark任务的API：
https://help.aliyun.com/document_detail/102357.html
AWS的通过Glue运行spark任务的API：
https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-jobs-job.html#aws-glue-api-jobs-job-CreateJob
### Motivation
目前Kyuubi Engine只能运行在Yarn或K8S上，这样在云原生的环境里要么需要申请EMR资源，要么需要申请K8S计算节点，这里存在两个问题：
1. EMR和K8S的资源不是弹性的，当任务少时，不能缩容以减少硬件成本，当任务多时，不能扩容，以提高计算速度。
2. 在云环境中，如果使用MaxCompute这样的弹性计算资源，JDBC只能使用Trino这样的交互式查询引擎，造成离线任务和交互式查询的SQL标准不完全一致。
### Describe the solution
通过将Kyuubi Engine运行在MaxCompute和Glue这种弹性Spark计算资源上，可以让离线批量任务和交互式查询共用相同的spark sql能力，也可以让计算资源有弹性，节省基础设施成本和运维成本。
### Additional context
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] badbye commented on issue #3409: [FEATURE] 让 Kyuubi Engine 跑在阿里MaxCompute或AWS Glue上
Posted by "badbye (via GitHub)" <gi...@apache.org>.
badbye commented on issue #3409:
URL: https://github.com/apache/kyuubi/issues/3409#issuecomment-1486402304
Seems it is using this [findLocalInetAddress](https://github.com/apache/kyuubi/blob/master/kyuubi-common/src/main/scala/org/apache/kyuubi/Utils.scala#L257) function to find the default IP.
Currently, there is no easy way to use the second NIC, am I right? @pan3793
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] github-actions[bot] commented on issue #3409: [FEATURE] 让 Kyuubi Engine 跑在阿里MaxCompute或AWS Glue上
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #3409:
URL: https://github.com/apache/incubator-kyuubi/issues/3409#issuecomment-1236813220
Hello @kevinclcn,
Thanks for finding the time to report the issue!
We really appreciate the community's efforts to improve Apache Kyuubi (Incubating).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] badbye commented on issue #3409: [FEATURE] 让 Kyuubi Engine 跑在阿里MaxCompute或AWS Glue上
Posted by "badbye (via GitHub)" <gi...@apache.org>.
badbye commented on issue #3409:
URL: https://github.com/apache/kyuubi/issues/3409#issuecomment-1485063573
> Have a quick look at the doc, I think Kyuubi should work out-of-box w/ MaxCompute, but not Glue. Since Kyuubi uses `spark-submit` to create spark engine app, technically, you can deploy Kyuubi in any environment as long as there is a runnable `spark-submit`(requires Spark 3.x) under `$SPARK_HOME/bin`
I'm trying to run Kyuubi with [Adb spark](https://help.aliyun.com/document_detail/471602.html) (it is similar to MaxCompute Spark), I got this error in Adb Spark:
> at org.apache.kyuubi.shade.org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)
23/03/27 20:46:27 ERROR ConnectionState: Connection timed out for connection string (beijing-datascience-dev-01:2181)
I'm using a standalone Kyuubi which has an EmbeddedZookeeper service, so the question is how to set the connection string of zookeeper to be the `ip:port` format instead of `hostname:port`? since the remote spark server does not know my hostname.
I've tried set [kyuubi.zookeeper.embedded.client.port.address](https://kyuubi.readthedocs.io/en/v1.6.1-incubating/deployment/settings.html#zookeeper) to be the public IP, it does not work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] pan3793 commented on issue #3409: [FEATURE] 让 Kyuubi Engine 跑在阿里MaxCompute或AWS Glue上
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on issue #3409:
URL: https://github.com/apache/kyuubi/issues/3409#issuecomment-1486190971
Kyuubi uses ISO-8601 standard duration format, please read comments of `java.time.Duration` to get more details.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] kevinclcn commented on issue #3409: [FEATURE] 让 Kyuubi Engine 跑在阿里MaxCompute或AWS Glue上
Posted by GitBox <gi...@apache.org>.
kevinclcn commented on issue #3409:
URL: https://github.com/apache/incubator-kyuubi/issues/3409#issuecomment-1237707901
>
Sure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] badbye commented on issue #3409: [FEATURE] 让 Kyuubi Engine 跑在阿里MaxCompute或AWS Glue上
Posted by "badbye (via GitHub)" <gi...@apache.org>.
badbye commented on issue #3409:
URL: https://github.com/apache/kyuubi/issues/3409#issuecomment-1487963314
Finally solved, I wrote a doc: https://gist.github.com/badbye/2618d6ef47a042427836d4ba9518e203
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] pan3793 commented on issue #3409: [FEATURE] 让 Kyuubi Engine 跑在阿里MaxCompute或AWS Glue上
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on issue #3409:
URL: https://github.com/apache/kyuubi/issues/3409#issuecomment-1486189663
And `kyuubi.session.engine.login.timeout = 30` means 30ms, I suppose you expected 30s not 30ms, the suggested format is PT30S
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] badbye commented on issue #3409: [FEATURE] 让 Kyuubi Engine 跑在阿里MaxCompute或AWS Glue上
Posted by "badbye (via GitHub)" <gi...@apache.org>.
badbye commented on issue #3409:
URL: https://github.com/apache/kyuubi/issues/3409#issuecomment-1486180297
After fixing the connection between the zookeeper and Adb Spark, I got a `connect timeout error` on the client side:
```
2023-03-28 11:47:11.728 INFO org.apache.kyuubi.ha.client.zookeeper.ZookeeperDiscoveryClient: Get service instance:21.25.1.59:45625 and version:Some(1.6.1-incubating) under /kyuubi_1.6.1-incubating_USER_SPARK_SQL/test/default
2023-03-28 11:47:11.768 ERROR org.apache.kyuubi.session.KyuubiSessionImpl: Opening engine [kyuubi_USER_SPARK_SQL_test_default_32adf216-e872-48a9-a87e-6789ef2d4a4c 21.25.1.59:45625] for test session failed
org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: connect timed out
at org.apache.thrift.transport.TSocket.open(TSocket.java:226) ~[libthrift-0.9.3.jar:0.9.3]
at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:266) ~[libthrift-0.9.3.jar:0.9.3]
at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37) ~[libthrift-0.9.3.jar:0.9.3]
at org.apache.kyuubi.client.KyuubiSyncThriftClient$.createTProtocol(KyuubiSyncThriftClient.scala:455) ~[kyuubi-server_2.12-1.6.1-incubating.jar:1.6.1-incubating]
at org.apache.kyuubi.client.KyuubiSyncThriftClient$.createClient(KyuubiSyncThriftClient.scala:471) ~[kyuubi-server_2.12-1.6.1-incubating.jar:1.6.1-incubating]
at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$openEngineSession$1(KyuubiSessionImpl.scala:128) ~[kyuubi-server_2.12-1.6.1-incubating.jar:1.6.1-incubating]
at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$openEngineSession$1$adapted(KyuubiSessionImpl.scala:113) ~[kyuubi-server_2.12-1.6.1-incubating.jar:1.6.1-incubating]
at org.apache.kyuubi.ha.client.DiscoveryClientProvider$.withDiscoveryClient(DiscoveryClientProvider.scala:36) ~[kyuubi-ha_2.12-1.6.1-incubating.jar:1.6.1-incubating]
at org.apache.kyuubi.session.KyuubiSessionImpl.openEngineSession(KyuubiSessionImpl.scala:113) ~[kyuubi-server_2.12-1.6.1-incubating.jar:1.6.1-incubating]
at org.apache.kyuubi.operation.LaunchEngine.$anonfun$runInternal$2(LaunchEngine.scala:49) ~[kyuubi-server_2.12-1.6.1-incubating.jar:1.6.1-incubating]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_271]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_271]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_271]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_271]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_271]
Caused by: java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_271]
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:476) ~[?:1.8.0_271]
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:218) ~[?:1.8.0_271]
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:200) ~[?:1.8.0_271]
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:394) ~[?:1.8.0_271]
at java.net.Socket.connect(Socket.java:606) ~[?:1.8.0_271]
at org.apache.thrift.transport.TSocket.open(TSocket.java:221) ~[libthrift-0.9.3.jar:0.9.3]
... 14 more
2023-03-28 11:47:11.774 INFO org.apache.curator.framework.imps.CuratorFrameworkImpl: backgroundOperationsLoop exiting
2023-03-28 11:47:11.777 INFO org.apache.zookeeper.ZooKeeper: Session: 0x10926b572df0001 closed
2023-03-28 11:47:11.777 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down for session: 0x10926b572df0001
2023-03-28 11:47:11.789 INFO org.apache.kyuubi.operation.LaunchEngine: Processing test's query[19ab56d1-a2eb-429e-a858-6d96b0ffdbbb]: RUNNING_STATE -> ERROR_STATE, time taken: 60.261 seconds
Error: org.apache.kyuubi.KyuubiSQLException: Error operating LaunchEngine: org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: connect timed out
at org.apache.thrift.transport.TSocket.open(TSocket.java:226)
at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:266)
at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
at org.apache.kyuubi.client.KyuubiSyncThriftClient$.createTProtocol(KyuubiSyncThriftClient.scala:455)
at org.apache.kyuubi.client.KyuubiSyncThriftClient$.createClient(KyuubiSyncThriftClient.scala:471)
at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$openEngineSession$1(KyuubiSessionImpl.scala:128)
at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$openEngineSession$1$adapted(KyuubiSessionImpl.scala:113)
at org.apache.kyuubi.ha.client.DiscoveryClientProvider$.withDiscoveryClient(DiscoveryClientProvider.scala:36)
at org.apache.kyuubi.session.KyuubiSessionImpl.openEngineSession(KyuubiSessionImpl.scala:113)
at org.apache.kyuubi.operation.LaunchEngine.$anonfun$runInternal$2(LaunchEngine.scala:49)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:476)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:218)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:200)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:394)
at java.net.Socket.connect(Socket.java:606)
at org.apache.thrift.transport.TSocket.open(TSocket.java:221)
... 14 more
at org.apache.kyuubi.KyuubiSQLException$.apply(KyuubiSQLException.scala:69)
at org.apache.kyuubi.operation.KyuubiOperation$$anonfun$onError$1.applyOrElse(KyuubiOperation.scala:75)
at org.apache.kyuubi.operation.KyuubiOperation$$anonfun$onError$1.applyOrElse(KyuubiOperation.scala:56)
at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:38)
at org.apache.kyuubi.operation.LaunchEngine.$anonfun$runInternal$2(LaunchEngine.scala:51)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: connect timed out
at org.apache.thrift.transport.TSocket.open(TSocket.java:226)
at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:266)
at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
at org.apache.kyuubi.client.KyuubiSyncThriftClient$.createTProtocol(KyuubiSyncThriftClient.scala:455)
at org.apache.kyuubi.client.KyuubiSyncThriftClient$.createClient(KyuubiSyncThriftClient.scala:471)
at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$openEngineSession$1(KyuubiSessionImpl.scala:128)
at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$openEngineSession$1$adapted(KyuubiSessionImpl.scala:113)
at org.apache.kyuubi.ha.client.DiscoveryClientProvider$.withDiscoveryClient(DiscoveryClientProvider.scala:36)
at org.apache.kyuubi.session.KyuubiSessionImpl.openEngineSession(KyuubiSessionImpl.scala:113)
at org.apache.kyuubi.operation.LaunchEngine.$anonfun$runInternal$2(LaunchEngine.scala:49)
... 5 more
Caused by: java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:476)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:218)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:200)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:394)
at java.net.Socket.connect(Socket.java:606)
at org.apache.thrift.transport.TSocket.open(TSocket.java:221)
... 14 more (state=,code=0)
Beeline version 1.6.1-incubating by Apache Kyuubi (Incubating)
```
any ideas to fix it? @pan3793
```
kyuubi.session.engine.login.timeout = 30
kyuubi.session.engine.alive.probe.interval = 30
kyuubi.session.engine.alive.timeout = 120
kyuubi.session.engine.alive.probe.enabled = true
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] pan3793 commented on issue #3409: [FEATURE] 让 Kyuubi Engine 跑在阿里MaxCompute或AWS Glue上
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on issue #3409:
URL: https://github.com/apache/kyuubi/issues/3409#issuecomment-1486188835
```
Get service instance:21.25.1.59:45625 and version:Some(1.6.1-incubating) under /kyuubi_1.6.1-incubating_USER_SPARK_SQL/test/default
```
Does ADB Spark allow Kyuubi Server to access the Driver throw IP directly?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] badbye commented on issue #3409: [FEATURE] 让 Kyuubi Engine 跑在阿里MaxCompute或AWS Glue上
Posted by "badbye (via GitHub)" <gi...@apache.org>.
badbye commented on issue #3409:
URL: https://github.com/apache/kyuubi/issues/3409#issuecomment-1486416448
> Yes, we need to enhance this part to make it more flexible, e.g. introduce an address-binding election strategy, it also helps for K8s environment.
Cool. I guess this is the last problem to make it work. I may not have the ability to contribute the code, but I'd like to write a doc. Let me know if there is any progress on this feature.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] pan3793 commented on issue #3409: [FEATURE] 让 Kyuubi Engine 跑在阿里MaxCompute或AWS Glue上
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on issue #3409:
URL: https://github.com/apache/kyuubi/issues/3409#issuecomment-1486406060
Yes, we need to enhance this part to make it more flexible, e.g. introduce an address-binding election strategy, it also helps for K8s environment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] badbye commented on issue #3409: [FEATURE] 让 Kyuubi Engine 跑在阿里MaxCompute或AWS Glue上
Posted by "badbye (via GitHub)" <gi...@apache.org>.
badbye commented on issue #3409:
URL: https://github.com/apache/kyuubi/issues/3409#issuecomment-1486197997
> ```
> Get service instance:21.25.1.59:45625 and version:Some(1.6.1-incubating) under /kyuubi_1.6.1-incubating_USER_SPARK_SQL/test/default
> ```
>
> Does ADB Spark allow Kyuubi Server to access the Driver through IP directly?
No, the Kyuubi server can not access this IP, I'll try to fix it.
I see, so I guess the whole workflow is:
1. Kyuubi server requests a spark session from the zookeeper, if no, start a Spark session vis spark-submit
2. after the spark session started, it register itself in the zookeeper
3. Kyuubi server finds spark session from zookeeper, tries to connect to the session directly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] pan3793 commented on issue #3409: [FEATURE] 让 Kyuubi Engine 跑在阿里MaxCompute或AWS Glue上
Posted by GitBox <gi...@apache.org>.
pan3793 commented on issue #3409:
URL: https://github.com/apache/incubator-kyuubi/issues/3409#issuecomment-1237646524
@kevinclcn would you like to try deploying Kyuubi on MaxCompute? and the docs are welcome.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] pan3793 commented on issue #3409: [FEATURE] 让 Kyuubi Engine 跑在阿里MaxCompute或AWS Glue上
Posted by GitBox <gi...@apache.org>.
pan3793 commented on issue #3409:
URL: https://github.com/apache/incubator-kyuubi/issues/3409#issuecomment-1236880984
Have a quick look at the doc, I think Kyuubi should work out-of-bix w/ MaxCompute, but not Glue. Since Kyuubi uses `spark-submit` to create spark engine app, technically, you can deploy Kyuubi in any environment as long as there is a runnable `spark-submit` under `$SPARK_HOME/bin`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] badbye commented on issue #3409: [FEATURE] 让 Kyuubi Engine 跑在阿里MaxCompute或AWS Glue上
Posted by "badbye (via GitHub)" <gi...@apache.org>.
badbye commented on issue #3409:
URL: https://github.com/apache/kyuubi/issues/3409#issuecomment-1486370602
> ```
> Get service instance:21.25.1.59:45625 and version:Some(1.6.1-incubating) under /kyuubi_1.6.1-incubating_USER_SPARK_SQL/test/default
> ```
>
> Does ADB Spark allow Kyuubi Server to access the Driver through IP directly?
Turns out the Adb Spark cluster has two NICs(Network Interface Cards), and the default NIC is used when the service starts. Is there a way to get it to boot and register to the second NIC?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] pan3793 commented on issue #3409: [FEATURE] 让 Kyuubi Engine 跑在阿里MaxCompute或AWS Glue上
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on issue #3409:
URL: https://github.com/apache/kyuubi/issues/3409#issuecomment-1485067402
the embedded zk is not recommend for production, it's designed to use for local testing, please deploy a dedicated zk first
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] badbye commented on issue #3409: [FEATURE] 让 Kyuubi Engine 跑在阿里MaxCompute或AWS Glue上
Posted by "badbye (via GitHub)" <gi...@apache.org>.
badbye commented on issue #3409:
URL: https://github.com/apache/kyuubi/issues/3409#issuecomment-1486198836
> And `kyuubi.session.engine.login.timeout = 30` means 30ms, I suppose you expect 30s not 30ms, the suggested format is PT30S
sorry, my bad. I've read the doc, just forget the unit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] pan3793 commented on issue #3409: [FEATURE] 让 Kyuubi Engine 跑在阿里MaxCompute或AWS Glue上
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on issue #3409:
URL: https://github.com/apache/kyuubi/issues/3409#issuecomment-1486218294
Yes, that's exactly how Kyuubi works, you got it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org