You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by "ruanwenjun (via GitHub)" <gi...@apache.org> on 2025/12/12 04:12:14 UTC
[I] [Feature][Task] Support config max task log size for per task [dolphinscheduler]
ruanwenjun opened a new issue, #17787:
URL: https://github.com/apache/dolphinscheduler/issues/17787
### Search before asking
- [x] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar feature requirement.
### Description
todo:
### Use case
_No response_
### Related issues
_No response_
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][Task] Support config max task log size for per task [dolphinscheduler]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #17787:
URL: https://github.com/apache/dolphinscheduler/issues/17787#issuecomment-3757523629
This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][Task] Support config max task log size for per task [dolphinscheduler]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #17787:
URL: https://github.com/apache/dolphinscheduler/issues/17787#issuecomment-4014992817
This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][Task] Support config max task log size for per task [dolphinscheduler]
Posted by "dill21yu (via GitHub)" <gi...@apache.org>.
dill21yu commented on issue #17787:
URL: https://github.com/apache/dolphinscheduler/issues/17787#issuecomment-3809318143
Currently, DolphinScheduler uses FileAppender for task logs without size limits, which can lead to large log files that consume excessive disk space.
Is this a feasible solution?
Replace FileAppender with RollingFileAppender and implement size-based rolling policy for task logs, similar to existing MASTERLOGFILE and WORKERLOGFILE configurations.
1. Configuration Changes:
dolphinscheduler-master/src/main/resources/logback-spring.xml
dolphinscheduler-worker/src/main/resources/logback-spring.xml
dolphinscheduler-standalone-server/src/main/resources/logback-spring.xml
Replace the TASKLOGFILE appender:
`<appender name="FILE-${taskInstanceLogFullPath}" class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>${taskInstanceLogFullPath}</file>
<rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
<fileNamePattern>${taskInstanceLogFullPath}.%d{yyyy-MM-dd_HH}.%i.log</fileNamePattern>
<maxFileSize>${task.log.max.file.size}</maxFileSize>
<maxHistory>${task.log.max.history}</maxHistory>
<totalSizeCap>${task.log.total.size.cap}</totalSizeCap>
<cleanHistoryOnStart>true</cleanHistoryOnStart>
</rollingPolicy>
<encoder>
<pattern>%date{yyyy-MM-dd HH:mm:ss.SSS} %-5level [%thread] - %message%n</pattern>
<charset>UTF-8</charset>
</encoder>
<append>true</append>
</appender>`
Add configurable properties:
`<property name="task.log.max.file.size" value="200MB"/>
<property name="task.log.max.history" value="3"/>
<property name="task.log.total.size.cap" value="600MB"/>`
2. Log Query Logic Updates
Modify logClientDelegate.getPartLogString() to handle multiple rolled files
Implement file discovery logic to find all rolled log files
Add cross-file line number calculation and content aggregation
Update log download functionality to aggregate all rolled files
And I am willing to submit a PR! @ruanwenjun @SbloodyS
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][Task] Support config max task log size for per task [dolphinscheduler]
Posted by "ruanwenjun (via GitHub)" <gi...@apache.org>.
ruanwenjun commented on issue #17787:
URL: https://github.com/apache/dolphinscheduler/issues/17787#issuecomment-3811090702
@dill21yu LGTM, but can it works? I am not sure if the usage is correct in logback.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][Task] Support config max task log size for per task [dolphinscheduler]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #17787: [Feature][Task] Support config max task log size for per task
URL: https://github.com/apache/dolphinscheduler/issues/17787
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org