You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2021/12/25 13:35:10 UTC
[GitHub] [dolphinscheduler] Narcasserun opened a new issue #7623: [Feature][plugin]Refactored data source plug-in loading and extension
Narcasserun opened a new issue #7623:
URL: https://github.com/apache/dolphinscheduler/issues/7623
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar feature requirement.
### Description
_No response_
### Use case
_No response_
### Related issues
_No response_
### Are you willing to submit a PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] KonsonChow edited a comment on issue #7623: [Feature][plugin]Refactored data source plug-in loading and extension
Posted by GitBox <gi...@apache.org>.
KonsonChow edited a comment on issue #7623:
URL: https://github.com/apache/dolphinscheduler/issues/7623#issuecomment-1019329335
Having experienced the need to be compatible with different versions of Hadoop clusters, with or without Kerberos authentication, it is really distressing. Dynamically loading UGI can solve this problem, and DS will be much more convenient to use. I think this is a very useful feature.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] Narcasserun commented on issue #7623: [Feature][plugin]Refactored data source plug-in loading and extension
Posted by GitBox <gi...@apache.org>.
Narcasserun commented on issue #7623:
URL: https://github.com/apache/dolphinscheduler/issues/7623#issuecomment-1001021808
This is a screenshot of my test after finishing the transformation locally
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] dailidong commented on issue #7623: [Feature][plugin]Refactored data source plug-in loading and extension
Posted by GitBox <gi...@apache.org>.
dailidong commented on issue #7623:
URL: https://github.com/apache/dolphinscheduler/issues/7623#issuecomment-1079962287
@zwZjut please give your opinion?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ruanwenjun commented on issue #7623: [Feature][plugin]Refactored data source plug-in loading and extension
Posted by GitBox <gi...@apache.org>.
ruanwenjun commented on issue #7623:
URL: https://github.com/apache/dolphinscheduler/issues/7623#issuecomment-1021873188
If this feature doesn't change the current SPI method, I think it's acceptable, but we need to make sure this will not make user confusion.
Waiting for community user to give more opinion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #7623: [Feature][plugin]Refactored data source plug-in loading and extension
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #7623:
URL: https://github.com/apache/dolphinscheduler/issues/7623#issuecomment-1001018807
Hi:
* Thank you for your feedback, we have received your issue, Please wait patiently for a reply.
* In order for us to understand your request as soon as possible, please provide detailed information、version or pictures.
* If you haven't received a reply for a long time, you can subscribe to the developer's email，Mail subscription steps reference https://dolphinscheduler.apache.org/en-us/community/development/subscribe.html ,Then write the issue URL in the email content and send question to dev@dolphinscheduler.apache.org.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] caishunfeng commented on issue #7623: [Feature][plugin]Refactored data source plug-in loading and extension
Posted by GitBox <gi...@apache.org>.
caishunfeng commented on issue #7623:
URL: https://github.com/apache/dolphinscheduler/issues/7623#issuecomment-1001185905
> Supports automatic acquisition of driver version and driver class
What's the solution?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] Narcasserun commented on issue #7623: [Feature][plugin]Refactored data source plug-in loading and extension
Posted by GitBox <gi...@apache.org>.
Narcasserun commented on issue #7623:
URL: https://github.com/apache/dolphinscheduler/issues/7623#issuecomment-1003023982
> > This is a big change, a new way of loading plugins has been reintroduced again.
> > I am not clear whether there are requirements of dynamic load the jar, it may need to discuss.
>
> I totally agree with @ruanwenjun about this point `whether there are requirements of dynamic load the jar`, in my humble opinion dynamically loading jars is less needed than you might thought in DolphinScheduler, although loading jars with some fancy technology might be cool at the first place, but they may be overused at most cases because do we really need this kind of stuff? It's better to discuss first whether we really need this considering this is big change and we just got rid of classloader things not too long ago.
I think we should pay more attention to what value this feature can bring to DS. It is wrong to blindly introduce new technologies and build cars behind closed doors. This feature is to isolate the configuration loading related to Hadoop, hive and Kerberos, which has nothing to do with the plug-in loading method of alarm, task and registry. Maybe it's limited from my own point of view @kezhenxu94 @ruanwenjun
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] kezhenxu94 commented on issue #7623: [Feature][plugin]Refactored data source plug-in loading and extension
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on issue #7623:
URL: https://github.com/apache/dolphinscheduler/issues/7623#issuecomment-1002891228
> This is a big change, a new way of loading plugins has been reintroduced again.
> I am not clear whether there are requirements of dynamic load the jar, it may need to discuss.
I totally agree with @ruanwenjun about this point `whether there are requirements of dynamic load the jar`, in my humble opinion dynamically loading jars is less needed than you might thought in DolphinScheduler, although loading jars with some fancy technology might be cool at the first place, but they may be overused at most cases because do we really need this kind of stuff? It's better to discuss first whether we really need this considering this is big change and we just got rid of classloader things not too long ago.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ruanwenjun commented on issue #7623: [Feature][plugin]Refactored data source plug-in loading and extension
Posted by GitBox <gi...@apache.org>.
ruanwenjun commented on issue #7623:
URL: https://github.com/apache/dolphinscheduler/issues/7623#issuecomment-1001165985
This is a big change, a new way of loading plugins has been reintroduced again.
I am not clear whether there are requirements of dynamic load the jar, it may need to discuss.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] KonsonChow commented on issue #7623: [Feature][plugin]Refactored data source plug-in loading and extension
Posted by GitBox <gi...@apache.org>.
KonsonChow commented on issue #7623:
URL: https://github.com/apache/dolphinscheduler/issues/7623#issuecomment-1019329335
Having experienced the need to be compatible with different versions of Hadoop clusters, with or without Kerberos authentication, it is really distressing. Dynamically loading UGI can solve this problem, and DS will be much more convenient to use. I think support is a useful feature.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] geosmart commented on issue #7623: [Feature][plugin]Refactored data source plug-in loading and extension
Posted by GitBox <gi...@apache.org>.
geosmart commented on issue #7623:
URL: https://github.com/apache/dolphinscheduler/issues/7623#issuecomment-1001263131
yarn,hdfs,kerberos all this config file can manager by `apiserver` ,
spark,flink can config default config on ui,
and different tenant has different config files. like this :
and the user can select config in the process definition
so the worker dynamic download config to run the task.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] dailidong removed a comment on issue #7623: [Feature][plugin]Refactored data source plug-in loading and extension
Posted by GitBox <gi...@apache.org>.
dailidong removed a comment on issue #7623:
URL: https://github.com/apache/dolphinscheduler/issues/7623#issuecomment-1079962287
@zwZjut please give your opinion?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org