[I] [Feature][Common] Support archiving task instance logs into zip and viewing archived logs online [dolphinscheduler]

["fatmanverse (via GitHub)" <gi...@apache.org>]

fatmanverse opened a new issue, #18420:
URL: https://github.com/apache/dolphinscheduler/issues/18420

   ### Search before asking
   
   - [X] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar feature requirement.
   
   ### Description
   
   Currently task instance / workflow instance logs are kept on the master and worker local disks forever. On long-running clusters the log base directory grows without bound and there is no built-in way to reclaim space or set a retention policy.
   
   This feature proposes an optional local task-log archiver:
   
   1. **Enable/disable archiving** — a master switch (`task-log-archive.enabled`), disabled by default so upgraded clusters keep their current behavior.
   2. **Archive logs older than a threshold** — daily log partitions older than `task-log-archive.archive-threshold` (default 30d) are compressed into a per-day zip under `{logBase}/archive/{yyyyMMdd}.zip`, and the source directory is removed to reclaim space. Optional expiry (`task-log-archive.expire-enabled` / `expire-threshold`) permanently deletes zips past a longer threshold; off by default so archiving alone never drops data.
   3. **View archived logs online** — the existing log-view path transparently falls back to reading the log entry out of its archive zip when the original file is gone, so no UI or API change is needed. Zip (not tar.gz) is used so a single entry can be random-read cheaply.
   
   The archiver runs as a plain daemon on both master and worker (each holds the task logs it physically wrote); it is not leader-gated. It lives in `dolphinscheduler-common` so master and worker share one implementation.
   
   ### Use case
   
   Operators running DolphinScheduler for months/years need a retention policy for instance logs: compress old logs to save disk, optionally expire very old archives, while still being able to open an archived log from the UI for auditing/troubleshooting.
   
   ### Related issues
   
   _No response_
   
   ### Are you willing to submit a PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Feature][Common] Support archiving task instance logs into zip and viewing archived logs online [dolphinscheduler]

["SbloodyS (via GitHub)" <gi...@apache.org>]

SbloodyS closed issue #18420: [Feature][Common] Support archiving task instance logs into zip and viewing archived logs online
URL: https://github.com/apache/dolphinscheduler/issues/18420


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Feature][Common] Support archiving task instance logs into zip and viewing archived logs online [dolphinscheduler]

["SbloodyS (via GitHub)" <gi...@apache.org>]

SbloodyS commented on issue #18420:
URL: https://github.com/apache/dolphinscheduler/issues/18420#issuecomment-4987508683

   Duplicated with #18295


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org