[GitHub] [dolphinscheduler] rickchengx commented on issue #13017: [Feature][Remote Logging] Add support for writing task logs to remote storage

[GitBox <gi...@apache.org>]

rickchengx commented on issue #13017:
URL: https://github.com/apache/dolphinscheduler/issues/13017#issuecomment-1341891161

   Here are some discussions on the weekly meeting
   
   **Q1: In the k8s environment, users can choose to mount persistent volumes (E.g., [OSS](https://help.aliyun.com/document_detail/130911.html)) to synchronize task logs to remote storage.**
   R1: This is indeed a way to synchronize logs to remote storage, and only need to mount persistent volumes (PV). But there are still some shortcomings as below:
   * **Efficiency**: Since the PV is connected to the remote storage, the speed of log writing will be reduced, which will further affect the task execution of the worker. On the contrary, uploading the task log to the remote storage asynchronously through the remote logging mechanism will not affect the execution of the task.
   * **Generality**: PV is not suitable for some remote storage, such as elasticsearch. And also it is not applicable to DS deployed in non-k8s environment.
   
   **Q2: Users can configure whether to use remote storage for task logs**
   R2: Yes, users can decide whether to enable log remote storage through configuration, and specify the corresponding configuration of remote logging.
   
   **Q3: The master-server also has task logs, which need to be uploaded to remote storage in a unified manner.**
   R3: Yes, users can set the master's task log related remote storage configuration in Master's configuration.
   
   **Q4: Is it possible to set the task log retention policy through the configuration supported by the remote storage itself?**
   R4: This is a good idea and it can simplify the design of remote logging, I'll look into it.
   
   Thanks again for all the suggestions at the weekly meeting, please correct me if I'm wrong.
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org