You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by "SEZ9 (via GitHub)" <gi...@apache.org> on 2024/08/10 05:59:56 UTC
[I] [Improvement][Module Name] DS should support providing direct deployment solutions on public cloud-managed Kubernetes platform [dolphinscheduler]
SEZ9 opened a new issue, #16434:
URL: https://github.com/apache/dolphinscheduler/issues/16434
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar feature requirement.
### Description
I strongly support the idea of DS enabling direct deployment on cloud-managed Kubernetes services, particularly Amazon EKS. Here are the reasons:
**Growing Cloud Adoption**: In today's rapidly evolving cloud computing landscape, over 45% of enterprises in China have adopted public cloud technologies as of 2023, with an annual growth rate of 5%. Managed Kubernetes services are a crucial part of this trend. According to Amazon Web Services, 82% of Kubernetes applications in the cloud run on AWS EKS. As a stable platform, DS should be compatible with such deployment solutions.
**Open Source and Compatibility:** Open source software is widely embraced because of its neutrality and compatibility, allowing for broad adoption. It should not be limited to supporting only certain open-source services while neglecting cloud-native managed services.
**Enhanced Flexibility and Cloud-Native Features**: While DS currently offers a Terraform Amazon EC2 deployment solution, this approach lacks the elasticity and full utilization of cloud-native features that an EKS deployment offers. For instance, serverless databases provide better elasticity for DS, and tools like Karpenter enable faster, more efficient scaling.
In conclusion, DS should support deployment across multiple platforms, including public cloud deployment solutions.
### Are you willing to submit a PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Improvement][Deployment] DS should support providing direct deployment solutions on public cloud-managed Kubernetes platform [dolphinscheduler]
Posted by "SbloodyS (via GitHub)" <gi...@apache.org>.
SbloodyS commented on issue #16434:
URL: https://github.com/apache/dolphinscheduler/issues/16434#issuecomment-2280072485
> DS is an open source component, like Flink and Doris. We only want to support open source k8s. Each public cloud has its own > customized implementation of k8s, and we don't provide deployment solutions for all of them. These should be implemented
> by the public cloud itself.
I have already express my my opinion in #16433. I'm -1 on this. Let's see others opinion. @ruanwenjun @Gallardot @EricGao888 @caishunfeng
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Improvement][Deployment] DS should support providing direct deployment solutions on public cloud-managed Kubernetes platform [dolphinscheduler]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on issue #16434:
URL: https://github.com/apache/dolphinscheduler/issues/16434#issuecomment-2282176744
Here's a simple summary in English of the key enhancements in this PR:
Provides users with a one-click Terraform solution for automatically deploying infrastructure and the DolphinScheduler system on AWS.
Integrates key cloud-native features to enhance the robustness of DolphinScheduler's cloud architecture, including managed EKS clusters, Karpenter for elastic scaling, Amazon Aurora Serverless v2 for automatic database scaling, EFS for persistent shared network storage, Amazon S3 for remote logs, dependency and attachment management, and AWS IAM authentication management.
Maintains compatibility with the existing DolphinScheduler/deploy/terraform/aws container features.
Commits to ongoing maintenance of the modules as DolphinScheduler and AWS services are upgraded.
Enhances the convenience and robustness of using DolphinScheduler on the cloud.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Improvement][Deployment] DS should support providing direct deployment solutions on public cloud-managed Kubernetes platform [dolphinscheduler]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on issue #16434:
URL: https://github.com/apache/dolphinscheduler/issues/16434#issuecomment-2280379958
This PR enhances the DS deployment in `dolphinscheduler/deploy/terraform/aws`. Originally, the deployment was on AWS EC2, but now it supports AWS EKS, with automatic integration of Amazon Aurora Serverless v2 for the database, EFS for persistent storage, S3 for remote logs, and automatic IAM role construction. It also reuses some open-source Kubernetes Helm charts. #16433
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Improvement][Deployment] DS should support providing direct deployment solutions on public cloud-managed Kubernetes platform [dolphinscheduler]
Posted by "EricGao888 (via GitHub)" <gi...@apache.org>.
EricGao888 commented on issue #16434:
URL: https://github.com/apache/dolphinscheduler/issues/16434#issuecomment-2287937629
IMHO - https://github.com/apache/dolphinscheduler/pull/16433#issuecomment-2287937234
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Improvement][Deployment] DS should support providing direct deployment solutions on public cloud-managed Kubernetes platform [dolphinscheduler]
Posted by "qingwli (via GitHub)" <gi...@apache.org>.
qingwli commented on issue #16434:
URL: https://github.com/apache/dolphinscheduler/issues/16434#issuecomment-2283420966
I was -1 on this. My idea is like @SbloodyS
If we want to run ds in eks, we should do some effects on aws part, like they can use some modules or images, like one-click start an eks with ds.
Not change some helm in ds part.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Improvement][Deployment] DS should support providing direct deployment solutions on public cloud-managed Kubernetes platform [dolphinscheduler]
Posted by "Gallardot (via GitHub)" <gi...@apache.org>.
Gallardot commented on issue #16434:
URL: https://github.com/apache/dolphinscheduler/issues/16434#issuecomment-2280273926
I reviewed this PR #16433 . What specific EKS capabilities is this PR trying to add? karpenter ?
The PR also copies an existing set of Helm charts. Are there any deficiencies with deploying directly using current Helm charts? Why is this new functionality needed? Why not enhance the existing capabilities?
I'm -1 on this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Improvement][Deployment] DS should support providing direct deployment solutions on public cloud-managed Kubernetes platform [dolphinscheduler]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 closed issue #16434: [Improvement][Deployment] DS should support providing direct deployment solutions on public cloud-managed Kubernetes platform
URL: https://github.com/apache/dolphinscheduler/issues/16434
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org