You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2022/05/29 16:03:09 UTC
[GitHub] [dolphinscheduler] EricGao888 opened a new issue, #10283: [Feature] Add connection center feature for DS
EricGao888 opened a new issue, #10283:
URL: https://github.com/apache/dolphinscheduler/issues/10283
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar feature requirement.
### Description
* Add `Connection Center` feature for Dolphin Scheduler so that plugins will be able to achieve connection information such as host, endpoint, credentials to external system and switch them easily without restarting DS.
### Use case
* For more information of this feature, we could refer to [Apache Airflow Connections Management](https://airflow.apache.org/docs/apache-airflow/stable/howto/connection.html)
### Related issues
_No response_
### Are you willing to submit a PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] EricGao888 commented on issue #10283: [Feature] Add connection center feature for DS
Posted by GitBox <gi...@apache.org>.
EricGao888 commented on issue #10283:
URL: https://github.com/apache/dolphinscheduler/issues/10283#issuecomment-1140478374
Could anyone please help add the `discussion` label to this issue? I think we need some discussions for the design of this feature : )
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] SbloodyS commented on issue #10283: [Feature] Add configuration center feature for DS
Posted by GitBox <gi...@apache.org>.
SbloodyS commented on issue #10283:
URL: https://github.com/apache/dolphinscheduler/issues/10283#issuecomment-1161206126
> Seems there would be two kinds of configurations if we implement this feature on the basis of current `data center`.
>
> 1. Connection related configurations for datasource, task plugins, etc.
> 2. Some common configurations which are in `common.properties` at present.
>
> BTW, we may still keep less-frequently-changed and high-risky configurations in `common.properties`.
Yes. I think the data source center should be part of the configuration center.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] EricGao888 commented on issue #10283: [Feature] Add configuration center feature for DS
Posted by GitBox <gi...@apache.org>.
EricGao888 commented on issue #10283:
URL: https://github.com/apache/dolphinscheduler/issues/10283#issuecomment-1161218977
Some ideas about user experience:
* We will have each task plugin bound with default connection, which corresponds to task types.
* We set default values in default connections so that new users to Dolphin Scheduler will not need to spend time figuring out what this `configuration center` does if they just try some basic stuff.
* Users will be able to see default example values in connection related configurations to help them understand how to manage connections.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] davidzollo commented on issue #10283: [Feature] Add connection center feature for DS
Posted by GitBox <gi...@apache.org>.
davidzollo commented on issue #10283:
URL: https://github.com/apache/dolphinscheduler/issues/10283#issuecomment-1159459401
good idea
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] liqingwang commented on issue #10283: [DSIP-19][Feature] Add connection center feature for DS
Posted by "liqingwang (via GitHub)" <gi...@apache.org>.
liqingwang commented on issue #10283:
URL: https://github.com/apache/dolphinscheduler/issues/10283#issuecomment-1550774321
Hi @EricGao888 , Could you help realize multiple resource centers?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] EricGao888 commented on issue #10283: [Feature] Add configuration center feature for DS
Posted by GitBox <gi...@apache.org>.
EricGao888 commented on issue #10283:
URL: https://github.com/apache/dolphinscheduler/issues/10283#issuecomment-1161203180
Seems there would be two kinds of connections if we implement this feature on the basis of current `data center`.
1. Connection related configurations for datasource, task plugins, etc.
2. Some common configurations which are in `common.properties` at present.
BTW, we may still keep less-frequently-changed and high-risky configurations in `common.properties`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] EricGao888 commented on issue #10283: [Feature] Add connection center feature for DS
Posted by GitBox <gi...@apache.org>.
EricGao888 commented on issue #10283:
URL: https://github.com/apache/dolphinscheduler/issues/10283#issuecomment-1161169462
> It seems that this is more like a configuration center in ds. WDYT? @EricGao888
@SbloodyS Exactly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] EricGao888 commented on issue #10283: [DSIP-19][Feature] Add connection center feature for DS
Posted by "EricGao888 (via GitHub)" <gi...@apache.org>.
EricGao888 commented on issue #10283:
URL: https://github.com/apache/dolphinscheduler/issues/10283#issuecomment-1550799019
> Hi @EricGao888 , Could you help realize multiple resource centers?
We have already removed singleton design for storage operators in backend. Therefore, it is possible to support multiple remote resource storages in resource center. But we need a good design for UI interactions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] SbloodyS commented on issue #10283: [Feature] Add configuration center feature for DS
Posted by GitBox <gi...@apache.org>.
SbloodyS commented on issue #10283:
URL: https://github.com/apache/dolphinscheduler/issues/10283#issuecomment-1161204260
> Could you please help add a `discussion` label for this issue? Thx @SbloodyS
Sure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #10283: [Feature] Add connection center feature for DS
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #10283:
URL: https://github.com/apache/dolphinscheduler/issues/10283#issuecomment-1140478154
Thank you for your feedback, we have received your issue, Please wait patiently for a reply.
* In order for us to understand your request as soon as possible, please provide detailed information、version or pictures.
* If you haven't received a reply for a long time, you can [join our slack](https://join.slack.com/t/asf-dolphinscheduler/shared_invite/zt-omtdhuio-_JISsxYhiVsltmC5h38yfw) and send your question to channel `#troubleshooting`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] EricGao888 commented on issue #10283: [Feature] Add connection center feature for DS
Posted by GitBox <gi...@apache.org>.
EricGao888 commented on issue #10283:
URL: https://github.com/apache/dolphinscheduler/issues/10283#issuecomment-1159460511
This feature would be something like extended `datasource` feature which we already have. The difference is that all task plugins besides SQL related ones could gain access to it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] EricGao888 commented on issue #10283: [Feature] Add configuration center feature for DS
Posted by GitBox <gi...@apache.org>.
EricGao888 commented on issue #10283:
URL: https://github.com/apache/dolphinscheduler/issues/10283#issuecomment-1161203644
Could you please help add a `discussion` label for this issue? Thx @SbloodyS
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] EricGao888 commented on issue #10283: [Feature] Add configuration center feature for DS
Posted by GitBox <gi...@apache.org>.
EricGao888 commented on issue #10283:
URL: https://github.com/apache/dolphinscheduler/issues/10283#issuecomment-1161220249
Rules for configuration update should be established at the very beginning to avoid confusion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] EricGao888 commented on issue #10283: [Feature] Add configuration center feature for DS
Posted by GitBox <gi...@apache.org>.
EricGao888 commented on issue #10283:
URL: https://github.com/apache/dolphinscheduler/issues/10283#issuecomment-1208878281
I will give a design later this month. For the first stage, we will focus on task plugin connections part.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] EricGao888 commented on issue #10283: [Feature] Add connection center feature for DS
Posted by GitBox <gi...@apache.org>.
EricGao888 commented on issue #10283:
URL: https://github.com/apache/dolphinscheduler/issues/10283#issuecomment-1159466474
> BTW, as discussed, we need some kind of `data masking` mechanism to hide sensitive data. Should we open another issue for `data masking`?
Add here: #10498
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] EricGao888 commented on issue #10283: [Feature] Add connection center feature for DS
Posted by GitBox <gi...@apache.org>.
EricGao888 commented on issue #10283:
URL: https://github.com/apache/dolphinscheduler/issues/10283#issuecomment-1159465799
BTW, as discussed, we need some kind of `data masking` mechanism to hide sensitive data. Should we open another issue for `data masking`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] SbloodyS commented on issue #10283: [Feature] Add connection center feature for DS
Posted by GitBox <gi...@apache.org>.
SbloodyS commented on issue #10283:
URL: https://github.com/apache/dolphinscheduler/issues/10283#issuecomment-1161168241
It seems that this is more like a configuration center in ds. WDYT? @EricGao888
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org