You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2022/11/28 05:07:43 UTC

[GitHub] [dolphinscheduler] rickchengx opened a new issue, #13017: [Feature][Remote Logging] Add support for writing task logs to remote storage

rickchengx opened a new issue, #13017:
URL: https://github.com/apache/dolphinscheduler/issues/13017

   ### Search before asking
   
   - [X] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar feature requirement.
   
   
   ### Description
   
   # Why remote logging?
   * Avoid task log loss after worker is torn down
   * Easier to obtain logs and troubleshoot after logs are aggregated in remote storage
   * Enhanced cloud-native support for DS
   
   # Feature Design
   
   ## Connect to different remote targets
   DS can support a variety of common remote storage, and has strong scalability to support other types of remote storage
   * S3
   * OSS
   * ElasticSearch
   * Azure Blob Storage
   * Google Cloud Storage
   * ...
   
   ## When to write logs to remote storage
   Like airflow, DS writes the task logs to remote storage after the task completes (success or fail).
   
   ## How to read logs
   Since the task log is stored in both the worker's local and remote storage, when the `api-server` needs to read the log of a certain task instance, it needs to determine the reading strategy.
   
   Airflow first tries to read the logs stored remotely, and if it fails, reads the local logs. But I prefer to try to read the local log first, and then read the remote log if the local log file does not exist.
   
   We could discuss this further.
   
   ## Log retention strategy
   
   For example, the maximum capacity of remote storage can be set, and old logs can be deleted by rolling.
   
   # Sub-tasks
   WIP
   
   # Ref
   * [Logging and Monitoring architecture in Airflow](https://airflow.apache.org/docs/apache-airflow/stable/logging-monitoring/logging-architecture.html)
   * [Remote logging in Airflow](https://airflow.apache.org/docs/apache-airflow-providers/core-extensions/logging.html)
   
   Any comments or suggestions are welcome.
   
   ### Use case
   
   Discussed above.
   
   ### Related issues
   
   * #8543
   * This issue is a sub-task of #12436
   
   ### Are you willing to submit a PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] Radeity commented on issue #13017: [Feature][Remote Logging] Add support for writing task logs to remote storage

Posted by GitBox <gi...@apache.org>.
Radeity commented on issue #13017:
URL: https://github.com/apache/dolphinscheduler/issues/13017#issuecomment-1328876157

   Hi, @rickchengx, i agree with you that we should keep consistency of local logs and remote logs. My central idea is to keep more useful logs after cleaning up, which can make DS different from Airflow. However, logs are widely used in practice for various maintenance activities, such as testing, failure diagnosis, and program comprehension, thus, my above suggestion to keep higher level log maybe immature. Anyway, thanks for your reply, hope my suggestion can give any inspiration!
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #13017: [Feature][Remote Logging] Add support for writing task logs to remote storage

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #13017:
URL: https://github.com/apache/dolphinscheduler/issues/13017#issuecomment-1328548911

   Thank you for your feedback, we have received your issue, Please wait patiently for a reply.
   * In order for us to understand your request as soon as possible, please provide detailed information、version or pictures.
   * If you haven't received a reply for a long time, you can [join our slack](https://s.apache.org/dolphinscheduler-slack) and send your question to channel `#troubleshooting`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] ruanwenjun commented on issue #13017: [DSIP-18][Remote Logging] Add support for writing task logs to remote storage

Posted by "ruanwenjun (via GitHub)" <gi...@apache.org>.
ruanwenjun commented on issue #13017:
URL: https://github.com/apache/dolphinscheduler/issues/13017#issuecomment-1694575970

   This seems already finished.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] fuchanghai commented on issue #13017: [Feature][Remote Logging] Add support for writing task logs to remote storage

Posted by GitBox <gi...@apache.org>.
fuchanghai commented on issue #13017:
URL: https://github.com/apache/dolphinscheduler/issues/13017#issuecomment-1333383859

   Good idea!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] Radeity commented on issue #13017: [Feature][Remote Logging] Add support for writing task logs to remote storage

Posted by GitBox <gi...@apache.org>.
Radeity commented on issue #13017:
URL: https://github.com/apache/dolphinscheduler/issues/13017#issuecomment-1328659415

   Good idea! One humble suggestion that **different log level** contributes to failure diagnosis differently, level of `error`, `warn`(high level) may contribute more than `info`, `debug`, `trace`(low level). Intuitively, it should better **keep more higher level log** and clean lower level log first when reach the maximum capacity of remote storage. Maybe we can separate different level of logs when flushing and support [log aggregation](https://bambrow.com/20221021-yarn-log-aggregation/#more) when reading logs like Yarn. WDYT?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] rickchengx commented on issue #13017: [Feature][Remote Logging] Add support for writing task logs to remote storage

Posted by GitBox <gi...@apache.org>.
rickchengx commented on issue #13017:
URL: https://github.com/apache/dolphinscheduler/issues/13017#issuecomment-1328731440

   > Good idea! One humble suggestion that **different log level** contributes to failure diagnosis differently, level of `error`, `warn`(high level) may contribute more than `info`, `debug`, `trace`(low level). Intuitively, it should better **keep more higher level log** and clean lower level log first when reach the maximum capacity of remote storage. Maybe we can separate different level of logs when flushing and support [log aggregation](https://bambrow.com/20221021-yarn-log-aggregation/#more) when reading logs like Yarn. WDYT?
   
   Hi, @Radeity , thanks a lot for your comment and suggestion.
   
   * This PR focus on writing task logs to remote storage after the task completes (E.g., it put the task log file to OSS). Also it reads the task log from the remote storage if the task log file does not exist on the local file system in worker.
   * So I think we should **keep the content consistency** of local logs and remote logs (E.g., the `log-2` and `log-3` in the image below) instead of aggregating logs on remote storage.
   * If the logs are aggregated and cleared according to the log level, users may be confused that he can only see part of the task logs.
   * So I think it is reasonable to clean up the oldest logs when the logs stored remotely reach the upper limit, just like `RollingFileAppender` in `logback`.
   
   <img width="630" alt="截屏2022-11-28 16 33 04" src="https://user-images.githubusercontent.com/38122586/204231822-b411664f-73c8-45a8-999a-397f85de8a71.png">
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] ruanwenjun closed issue #13017: [DSIP-18][Remote Logging] Add support for writing task logs to remote storage

Posted by "ruanwenjun (via GitHub)" <gi...@apache.org>.
ruanwenjun closed issue #13017: [DSIP-18][Remote Logging] Add support for writing task logs to remote storage
URL: https://github.com/apache/dolphinscheduler/issues/13017


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] rickchengx commented on issue #13017: [Feature][Remote Logging] Add support for writing task logs to remote storage

Posted by GitBox <gi...@apache.org>.
rickchengx commented on issue #13017:
URL: https://github.com/apache/dolphinscheduler/issues/13017#issuecomment-1341891161

   Here are some discussions on the weekly meeting
   
   **Q1: In the k8s environment, users can choose to mount persistent volumes (E.g., [OSS](https://help.aliyun.com/document_detail/130911.html)) to synchronize task logs to remote storage.**
   R1: This is indeed a way to synchronize logs to remote storage, and only need to mount persistent volumes (PV). But there are still some shortcomings as below:
   * **Efficiency**: Since the PV is connected to the remote storage, the speed of log writing will be reduced, which will further affect the task execution of the worker. On the contrary, uploading the task log to the remote storage asynchronously through the remote logging mechanism will not affect the execution of the task.
   * **Generality**: PV is not suitable for some remote storage, such as elasticsearch. And also it is not applicable to DS deployed in non-k8s environment.
   
   **Q2: Users can configure whether to use remote storage for task logs**
   R2: Yes, users can decide whether to enable log remote storage through configuration, and specify the corresponding configuration of remote logging.
   
   **Q3: The master-server also has task logs, which need to be uploaded to remote storage in a unified manner.**
   R3: Yes, users can set the master's task log related remote storage configuration in Master's configuration.
   
   **Q4: Is it possible to set the task log retention policy through the configuration supported by the remote storage itself?**
   R4: This is a good idea and it can simplify the design of remote logging, I'll look into it.
   
   Thanks again for all the suggestions at the weekly meeting, please correct me if I'm wrong.
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org