You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@kyuubi.apache.org by "davidshtian (via GitHub)" <gi...@apache.org> on 2023/03/06 06:52:09 UTC
[GitHub] [kyuubi] davidshtian opened a new issue, #4458: [FEATURE] Support Amazon EMR Serverless on AWS
davidshtian opened a new issue, #4458:
URL: https://github.com/apache/kyuubi/issues/4458
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [X] I have searched in the [issues](https://github.com/apache/kyuubi/issues?q=is%3Aissue) and found no similar issues.
### Describe the feature
Support [Amazon EMR Serverless](https://aws.amazon.com/emr/serverless/) as Kyuubi Spark Engine to minimize the operation cost and implement real serverless Spark SQL goal. Amazon EMR Serverless is not supported yet as Amazon EMR Serverless has no JDBC connection.
### Motivation
Implement real serverless Spark SQL target on AWS cloud.
### Describe the solution
Amazon EMR Serverless makes it easy for users to run Spark without configuring, managing, and scaling clusters or servers.
### Additional context
_No response_
### Are you willing to submit PR?
- [ ] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to improve.
- [X] No. I cannot submit a PR at this time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [TASK][MEDIUM] Support Amazon EMR Serverless on AWS [kyuubi]
Posted by "zhaohehuhu (via GitHub)" <gi...@apache.org>.
zhaohehuhu commented on issue #4458:
URL: https://github.com/apache/kyuubi/issues/4458#issuecomment-2055451519
@pan3793 plz assign it to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [FEATURE] Support Amazon EMR Serverless on AWS [kyuubi]
Posted by "PauloMigAlmeida (via GitHub)" <gi...@apache.org>.
PauloMigAlmeida commented on issue #4458:
URL: https://github.com/apache/kyuubi/issues/4458#issuecomment-1965327146
@pan3793, @yaooqinn
> I'm not an AWS user, seems that [localstack](https://github.com/localstack/localstack) also does not support AWS EMR Serverless?
You are right, localstack doesn't support the endpoints for dealing with EMR serverless yet
> I'm afraid that functionality without CI verification is fragile.
I agree. Does the apache foundation have AWS accounts that can be used for CI/CD purposes? If so, that would be the fastest way to address this as I think we may not be the first one needing that type of integration tests.
If it doesn't, you may want to try the AWS promotional credits for Open Source projects. More info at https://aws.amazon.com/blogs/opensource/aws-promotional-credits-open-source-projects/
Regardless of the option you go with, count on me for helping with the AWS implementation details (IAM permissions, service configuration, sdk, cli options and so on) - that can speed up the time machine for the development of this feature.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [TASK][MEDIUM] Support Amazon EMR Serverless on AWS [kyuubi]
Posted by "zhaohehuhu (via GitHub)" <gi...@apache.org>.
zhaohehuhu commented on issue #4458:
URL: https://github.com/apache/kyuubi/issues/4458#issuecomment-2058436771
Thanks. It looks like it's hard for Kyuubi service running on EC2 to access Amazon EMR Serverless Spark. I will discuss it with @pan3793.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [FEATURE] Support Amazon EMR Serverless on AWS [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on issue #4458:
URL: https://github.com/apache/kyuubi/issues/4458#issuecomment-1970907608
cc @zhaohehuhu, you may be interested in this feature
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] yaooqinn commented on issue #4458: [FEATURE] Support Amazon EMR Serverless on AWS
Posted by "yaooqinn (via GitHub)" <gi...@apache.org>.
yaooqinn commented on issue #4458:
URL: https://github.com/apache/kyuubi/issues/4458#issuecomment-1463345361
> EMR Serverless has no JDBC connection and it uses [AWS API](https://docs.aws.amazon.com/emr/latest/EMR-Serverless-UserGuide/jobs-spark.html) to submit the job
Sorry for being late.
Never used AWS. Correct me if I am wrong, do you mean that an AWS EMR cluster does not support `spark-submit`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [FEATURE] Support Amazon EMR Serverless on AWS [kyuubi]
Posted by "PauloMigAlmeida (via GitHub)" <gi...@apache.org>.
PauloMigAlmeida commented on issue #4458:
URL: https://github.com/apache/kyuubi/issues/4458#issuecomment-1963178638
Hi @yaooqinn,
> do you mean that an AWS EMR cluster does not support spark-submit?
EMR (on EC2) does support submitting via `spark-submit` utility. However, EMR-Serverless do not.
For EMR-Serverless, one can submit jobs by either using of the AWS' SDK or via AWSCli.
It looks something like this:
```bash
aws emr-serverless start-job-run \
--application-id <EMR_Severless_App_Id> \
--execution-role-arn arn:aws:iam::012345678901:role/my-cool-emr-exec-role \
--job-driver 'sparkSubmit={entryPoint=s3://my-bucket/script_to_be_execeuted.py}' \
--configuration-overrides '{"monitoringConfiguration": {"managedPersistenceMonitoringConfiguration": {"enabled": true}, "cloudWatchLoggingConfiguration": {"enabled": true, "logGroupName": "/aws/emr-serverless/my-logs"}}}'
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [FEATURE] Support Amazon EMR Serverless on AWS [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on issue #4458:
URL: https://github.com/apache/kyuubi/issues/4458#issuecomment-1963296527
To support AWS EMR Serverless Spark, we need to implement the following interface in Kyuubi
```
org.apache.kyuubi.engine.ApplicationOperation (for querying and canceling job)
org.apache.kyuubi.engine.ProcBuilder (for submitting job)
```
According to the [EMR Docs](https://docs.aws.amazon.com/emr/latest/EMR-Serverless-UserGuide/setting-up.html), I think we can use the CLI `aws emr-serverless` to implement that.
One concern is the integration tests, I'm not an AWS user, seems that [localstack](https://github.com/localstack/localstack) also does not support AWS EMR Serverless? I'm afraid that functionality without CI verification is fragile.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [FEATURE] Support Amazon EMR Serverless on AWS [kyuubi]
Posted by "zhaohehuhu (via GitHub)" <gi...@apache.org>.
zhaohehuhu commented on issue #4458:
URL: https://github.com/apache/kyuubi/issues/4458#issuecomment-2017598989
> @zhaohehuhu If that helps, this is the terraform code that can provision an EMR Serverless cluster with the right permissions https://gist.github.com/PauloMigAlmeida/5cebf3efcd0f105d73646a6a9e8cc2f3
>
> Instructions on how to deploy and run it are in the gist too.
Thanks. may contact you if needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [FEATURE] Support Amazon EMR Serverless on AWS [kyuubi]
Posted by "zhaohehuhu (via GitHub)" <gi...@apache.org>.
zhaohehuhu commented on issue #4458:
URL: https://github.com/apache/kyuubi/issues/4458#issuecomment-1972565522
Yup. I'm going to implement this feature.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [TASK][MEDIUM] Support Amazon EMR Serverless on AWS [kyuubi]
Posted by "zhaohehuhu (via GitHub)" <gi...@apache.org>.
zhaohehuhu commented on issue #4458:
URL: https://github.com/apache/kyuubi/issues/4458#issuecomment-2058245035
Thanks @PauloMigAlmeida. The EMR serverless runs on a VPC mannered by AWS, I just wonder it is possbile for Kyuubi Service talk to Spark Engine in EMR serverless through Thrift Protocol.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [TASK][MEDIUM] Support Amazon EMR Serverless on AWS [kyuubi]
Posted by "PauloMigAlmeida (via GitHub)" <gi...@apache.org>.
PauloMigAlmeida commented on issue #4458:
URL: https://github.com/apache/kyuubi/issues/4458#issuecomment-2060250951
@pan3793 Seems to be that we don't have thrift server running on those nodes for the serverless offering.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [TASK][MEDIUM] Support Amazon EMR Serverless on AWS [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on issue #4458:
URL: https://github.com/apache/kyuubi/issues/4458#issuecomment-2060282943
@PauloMigAlmeida connect initiate from Kyuubi server to Spark driver. The detailed steps are:
1. when a new connection comes in, Kyuubi looks up Zookeeper to find a reusable Spark application. If not found, try to perform a `spark-submit` to launch a new Spark app (a Kyuubi-customized Spark app, called Kyuubi Spark SQL engine).
2. after the Spark driver starts, it launches a Thrift RPC server, and registers itself to Zookeeper, so that the Kyuubi server knows how to connect to this RPC server.
3. Kyuubi server connects to the Spark driver and forwards queries to it.
4. Spark application self terminates (also deregister from zookeeper) after idle(no active connections) timeout
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [TASK][MEDIUM] Support Amazon EMR Serverless on AWS [kyuubi]
Posted by "zhaohehuhu (via GitHub)" <gi...@apache.org>.
zhaohehuhu commented on issue #4458:
URL: https://github.com/apache/kyuubi/issues/4458#issuecomment-2058153056
@PauloMigAlmeida I deployed a kyuubi server on EC2. When kyuubi server talks to Spark engine in EMR sververless, it always says connection timeout. Kyuubi server and EMR sververless already are in the same VPC, how should it be? Do you have any idea about it ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] github-actions[bot] commented on issue #4458: [FEATURE] Support Amazon EMR Serverless on AWS
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #4458:
URL: https://github.com/apache/kyuubi/issues/4458#issuecomment-1455547898
Hello @davidshtian,
Thanks for finding the time to report the issue!
We really appreciate the community's efforts to improve Apache Kyuubi.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [FEATURE] Support Amazon EMR Serverless on AWS [kyuubi]
Posted by "PauloMigAlmeida (via GitHub)" <gi...@apache.org>.
PauloMigAlmeida commented on issue #4458:
URL: https://github.com/apache/kyuubi/issues/4458#issuecomment-2010717343
@zhaohehuhu just checking in to see if you need any help with the AWS side of things
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [FEATURE] Support Amazon EMR Serverless on AWS [kyuubi]
Posted by "zhaohehuhu (via GitHub)" <gi...@apache.org>.
zhaohehuhu commented on issue #4458:
URL: https://github.com/apache/kyuubi/issues/4458#issuecomment-2014211712
> @zhaohehuhu just checking in to see if you need any help with the AWS side of things
Thanks. It's going well so far. I already finished the draft code and decided to do a round of test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [TASK][MEDIUM] Support Amazon EMR Serverless on AWS [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on issue #4458:
URL: https://github.com/apache/kyuubi/issues/4458#issuecomment-2060261713
@PauloMigAlmeida Kyuubi use thrift as internal RPC protocol for Kyuubi server and Spark driver, it will auto bootstrap a thrift server on the Spark driver. So the question is, is Spark Servless allows Thrift(kind of a TCP-based protocol) traffic between outside and inside, I suppose it should work, the Jupyter Notebook case runs in similiar way.
https://docs.aws.amazon.com/emr/latest/EMR-Serverless-UserGuide/interactive-workloads.html
I have a offline talk with @zhaohehuhu, according his feedback, the driver successfully launched Thrift RPC server, and registered it to Zookeeper, so Kyuubi server got the Thrift RPC server address but can NOT establish the connection. We have reported similar issues on other public cloud vendor, it caused by dual NICs https://github.com/apache/kyuubi/issues/6296, I'm not sure what's exact issue on AWS EMR Serverless
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [TASK][MEDIUM] Support Amazon EMR Serverless on AWS [kyuubi]
Posted by "PauloMigAlmeida (via GitHub)" <gi...@apache.org>.
PauloMigAlmeida commented on issue #4458:
URL: https://github.com/apache/kyuubi/issues/4458#issuecomment-2060238686
@zhaohehuhu I got hold of a EMR Serverless specialist internally. Thrift communication isn't possible at this moment on EMR Serverless =/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [TASK][MEDIUM] Support Amazon EMR Serverless on AWS [kyuubi]
Posted by "PauloMigAlmeida (via GitHub)" <gi...@apache.org>.
PauloMigAlmeida commented on issue #4458:
URL: https://github.com/apache/kyuubi/issues/4458#issuecomment-2058211295
@zhaohehuhu EMRServerless doesn't run a customer-managed VPC. It's accessible via an API that should be invoked using one of the AWS SDKs instead.
API method: https://docs.aws.amazon.com/emr-serverless/latest/APIReference/API_StartJobRun.html
AWS SDK for Java: https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/services/emrserverless/EmrServerlessClient.html#startJobRun(software.amazon.awssdk.services.emrserverless.model.StartJobRunRequest)
PS: It requires that an EMRServerless application is created.
Does this help?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [FEATURE] Support Amazon EMR Serverless on AWS [kyuubi]
Posted by "yaooqinn (via GitHub)" <gi...@apache.org>.
yaooqinn commented on issue #4458:
URL: https://github.com/apache/kyuubi/issues/4458#issuecomment-1965666668
It looks like ASF Infra doesn't have AWS resources for CI/CD. https://infra.apache.org/build-supported-services.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [FEATURE] Support Amazon EMR Serverless on AWS [kyuubi]
Posted by "yaooqinn (via GitHub)" <gi...@apache.org>.
yaooqinn commented on issue #4458:
URL: https://github.com/apache/kyuubi/issues/4458#issuecomment-1965725961
> Another option would be to mock responses expected from the AWS services involved.
This is a necessary step for local dev. AWS promotional credits might be necessary for setting the integration tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] yaooqinn commented on issue #4458: [FEATURE] Support Amazon EMR Serverless on AWS
Posted by "yaooqinn (via GitHub)" <gi...@apache.org>.
yaooqinn commented on issue #4458:
URL: https://github.com/apache/kyuubi/issues/4458#issuecomment-1455715757
Does deploying Kyuubi on Amazon EMR satisfy this？ AFAIK, cloud vendors like Tencent cloud and Aliyun, who provide similar EMR services, have provided JDBC though Kyuubi
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] davidshtian commented on issue #4458: [FEATURE] Support Amazon EMR Serverless on AWS
Posted by "davidshtian (via GitHub)" <gi...@apache.org>.
davidshtian commented on issue #4458:
URL: https://github.com/apache/kyuubi/issues/4458#issuecomment-1455769600
> Does deploying Kyuubi on Amazon EMR satisfy this？ AFAIK, cloud vendors like Tencent cloud and Aliyun, who provide similar EMR services, have provided JDBC though Kyuubi
Thanks for your response~
Kyuubi can be deployed on AWS EMR cluster mode (EMR on EC2), but it still need to manage and operate the cluster, while EMR Serverless is fully serverless and managed bringing more flexibility and it could handle different scenarios. EMR Serverless has no JDBC connection and it uses [AWS API](https://docs.aws.amazon.com/emr/latest/EMR-Serverless-UserGuide/jobs-spark.html) to submit the job, it would be better to have the support to adapt EMR Serverless to Kyuubi. Thanks~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [TASK][MEDIUM] Support Amazon EMR Serverless on AWS [kyuubi]
Posted by "PauloMigAlmeida (via GitHub)" <gi...@apache.org>.
PauloMigAlmeida commented on issue #4458:
URL: https://github.com/apache/kyuubi/issues/4458#issuecomment-2060316500
@pan3793 I was afraid you were going say that (Kyuubi server to Spark driver flow).
I triple-checked that internally, at the moment inbound connections to EMR Serverless are not possible. I can't share the nitty-gritty of why but it seems to be by design.
I just put together a Product Feature Request on that to the service team so AWS is aware that this is something customers would want though.
Implementation-wise, are there any alternatives to circumvent that limitation?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [TASK][MEDIUM] Support Amazon EMR Serverless on AWS [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on issue #4458:
URL: https://github.com/apache/kyuubi/issues/4458#issuecomment-2060245772
@PauloMigAlmeida do you know the exact restriction? TCP inbound traffic or something?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [TASK][MEDIUM] Support Amazon EMR Serverless on AWS [kyuubi]
Posted by "zhaohehuhu (via GitHub)" <gi...@apache.org>.
zhaohehuhu commented on issue #4458:
URL: https://github.com/apache/kyuubi/issues/4458#issuecomment-2060330663
The outbound traffic to EMR Serverless can be solved by the VPC, but the inbound traffic seems to be a problem now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [FEATURE] Support Amazon EMR Serverless on AWS [kyuubi]
Posted by "PauloMigAlmeida (via GitHub)" <gi...@apache.org>.
PauloMigAlmeida commented on issue #4458:
URL: https://github.com/apache/kyuubi/issues/4458#issuecomment-1965710057
Another option would be to mock responses expected from the AWS services involved.
I've seen that done before in other project in which localstack didn't support the service required. Would it address the CI tests concern?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [TASK][MEDIUM] Support Amazon EMR Serverless on AWS [kyuubi]
Posted by "PauloMigAlmeida (via GitHub)" <gi...@apache.org>.
PauloMigAlmeida commented on issue #4458:
URL: https://github.com/apache/kyuubi/issues/4458#issuecomment-2060274272
@pan3793 got you point now.
Does the thrift communication initiate from the EMR serverless to the kyuubi server? Or is it the other way around?
In case, it's the former:
Out of curiosity, what's the security groups rules for both kyuubi server and EMR serverless (with VPC)?
I'm aware that EMR serverless can establish connections within a VPC
https://docs.aws.amazon.com/emr/latest/EMR-Serverless-UserGuide/vpc-access.html
The example above was for integration with databases but the set up should be relatively the same if the flow is from EMR serverless to kyuubi server
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [TASK][MEDIUM] Support Amazon EMR Serverless on AWS [kyuubi]
Posted by "PauloMigAlmeida (via GitHub)" <gi...@apache.org>.
PauloMigAlmeida commented on issue #4458:
URL: https://github.com/apache/kyuubi/issues/4458#issuecomment-2058381647
@zhaohehuhu I'm almost certain that it isn't supported but let me check that internally first and I will come back to you with an answer tomorrow.
On a separate note, if communicating via thrift isn't possible, is there any alternative that could be explored instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [TASK][MEDIUM] Support Amazon EMR Serverless on AWS [kyuubi]
Posted by "sirajulm (via GitHub)" <gi...@apache.org>.
sirajulm commented on issue #4458:
URL: https://github.com/apache/kyuubi/issues/4458#issuecomment-2231459857
There could be a workaround using external Hive metastore using AWS RDS or reusing Thrift server for existing EMR. This would mean additional resources need to be created and would make it complex to test.
https://docs.aws.amazon.com/emr/latest/EMR-Serverless-UserGuide/metastore-config.html#external-metastore
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [FEATURE] Support Amazon EMR Serverless on AWS [kyuubi]
Posted by "PauloMigAlmeida (via GitHub)" <gi...@apache.org>.
PauloMigAlmeida commented on issue #4458:
URL: https://github.com/apache/kyuubi/issues/4458#issuecomment-2017144555
@zhaohehuhu If that helps, this is the terraform code that can provision an EMR Serverless cluster with the right permissions https://gist.github.com/PauloMigAlmeida/5cebf3efcd0f105d73646a6a9e8cc2f3
Instruction on how to deploy it and run it is in the gist too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [FEATURE] Support Amazon EMR Serverless on AWS [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on issue #4458:
URL: https://github.com/apache/kyuubi/issues/4458#issuecomment-1963298417
BTW, does GCP and Azure have similar services?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [FEATURE] Support Amazon EMR Serverless on AWS [kyuubi]
Posted by "yaooqinn (via GitHub)" <gi...@apache.org>.
yaooqinn commented on issue #4458:
URL: https://github.com/apache/kyuubi/issues/4458#issuecomment-1963234634
It looks like we need a specific version of `the org.apache.kyuubi.engine.ProcBuilder` for was. also cc @pan3793
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org