You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2022/02/25 09:51:00 UTC
[GitHub] [dolphinscheduler] EricGao888 opened a new issue #8543: [Feature][server] Add support for logging into remote storage
EricGao888 opened a new issue #8543:
URL: https://github.com/apache/dolphinscheduler/issues/8543
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar feature requirement.
### Description
- Add support for remote logging so that the tasks logs could be preserved in remote storage service like AWS S3 or Alibaba-Cloud OSS.
### Use case
- Currently, it seems dolphin scheduler does not support logging into remote storage such as AWS S3, Alibaba-Cloud OSS, etc.
- If dolphin scheduler is deployed on a remote cluster like EMR and the cluster is torn down or there's something wrong with the workers, users will probably lose the logs of previous executed tasks.
### Related issues
_No response_
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] EricGao888 commented on issue #8543: [Feature][server] Add support for logging into remote storage
Posted by GitBox <gi...@apache.org>.
EricGao888 commented on issue #8543:
URL: https://github.com/apache/dolphinscheduler/issues/8543#issuecomment-1060141348
> good suggestion, Can you give a design to support the remote logging system?
@dailidong I will try to give a design this week, thx : )
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] EricGao888 edited a comment on issue #8543: [Feature][server] Add support for logging into remote storage
Posted by GitBox <gi...@apache.org>.
EricGao888 edited a comment on issue #8543:
URL: https://github.com/apache/dolphinscheduler/issues/8543#issuecomment-1053846549
@caishunfeng @dailidong Another question related to this issue is how does ds deal with workers' logs? If not logging into remote storage, how do we keep large amount of logs produced by workers? Is there a best practice for this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] gaojun2048 commented on issue #8543: [Feature][server] Add support for logging into remote storage
Posted by GitBox <gi...@apache.org>.
gaojun2048 commented on issue #8543:
URL: https://github.com/apache/dolphinscheduler/issues/8543#issuecomment-1067627365
Yes, logging into remote storage is a good idea when worker is deploy in cloud.
We also need to consider whether the log target is global or whether different workflows can config they own log target.
If dolphin scheduler needs to run on hybrid cloud in the future, I think we should configure its own log target for each workflow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] EricGao888 commented on issue #8543: [Feature][server] Add support for logging into remote storage
Posted by GitBox <gi...@apache.org>.
EricGao888 commented on issue #8543:
URL: https://github.com/apache/dolphinscheduler/issues/8543#issuecomment-1053846549
@caishunfeng @dailidong Another question related to this issue is how does ds deal with workers' logs? If not logging into remote devices, how do we keep large amount of logs produced by workers? Is there a best practice for this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] dailidong commented on issue #8543: [Feature][server] Add support for logging into remote storage
Posted by GitBox <gi...@apache.org>.
dailidong commented on issue #8543:
URL: https://github.com/apache/dolphinscheduler/issues/8543#issuecomment-1050786473
good suggestion, Can you give a design to support the remote logging system?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] EricGao888 commented on issue #8543: [Feature][server] Add support for logging into remote storage
Posted by GitBox <gi...@apache.org>.
EricGao888 commented on issue #8543:
URL: https://github.com/apache/dolphinscheduler/issues/8543#issuecomment-1060141848
> > @caishunfeng @dailidong Another question related to this issue is how does ds deal with workers' logs? If not logging into remote storage, how do we keep large amount of logs produced by workers? Is there a best practice for this?
>
> Now DS doesn't specifically dealt with task log file, and the user handles it by himself, which is unfriendly. It is a goold idea to write the task log into remote storage when keep running large amout tasks. And I think the two point:
>
> 1. How to write files? Specified by configuration or config in the process definition UI?
> 2. How to read compatible local files and remote files?
>
> If you have some good idea, please let me know.
Will take these points into consideration. Thx for the suggestions!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] caishunfeng commented on issue #8543: [Feature][server] Add support for logging into remote storage
Posted by GitBox <gi...@apache.org>.
caishunfeng commented on issue #8543:
URL: https://github.com/apache/dolphinscheduler/issues/8543#issuecomment-1053948453
> @caishunfeng @dailidong Another question related to this issue is how does ds deal with workers' logs? If not logging into remote storage, how do we keep large amount of logs produced by workers? Is there a best practice for this?
Now DS doesn't specifically dealt with task log file, and the user handles it by himself, which is unfriendly.
It is a goold idea to write the task log into remote storage when keep running large amout tasks. And I think the two point:
1. How to write files? Specified by configuration or config in the process definition UI?
2. How to read compatible local files and remote files?
If you have some good idea, please let me know.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #8543: [Feature][server] Add support for logging into remote storage
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #8543:
URL: https://github.com/apache/dolphinscheduler/issues/8543#issuecomment-1050706989
Hi:
* Thank you for your feedback, we have received your issue, Please wait patiently for a reply.
* In order for us to understand your request as soon as possible, please provide detailed information、version or pictures.
* If you haven't received a reply for a long time, you can subscribe to the developer's email，Mail subscription steps reference https://dolphinscheduler.apache.org/en-us/community/development/subscribe.html ,Then write the issue URL in the email content and send question to dev@dolphinscheduler.apache.org.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] caishunfeng commented on issue #8543: [Feature][server] Add support for logging into remote storage
Posted by GitBox <gi...@apache.org>.
caishunfeng commented on issue #8543:
URL: https://github.com/apache/dolphinscheduler/issues/8543#issuecomment-1050709729
Is it necessary to add thie feature in DS?
Is it possible to use log monitoring synchronization to do it? Similar to Alibaba Cloud's SLS
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org