You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "freemandealer (via GitHub)" <gi...@apache.org> on 2023/11/23 09:38:42 UTC
[I] [Enhancement] limit the amount of error_log when load to save disks [doris]
freemandealer opened a new issue, #27481:
URL: https://github.com/apache/doris/issues/27481
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues.
### Description
Description in English:
In the load process, if there are problems with the original data, we will store the error data in an error_log file on the disk for subsequent debugging. However, if there are many error data, it will occupy a lot of disk space. Now we want to limit the number of error data that is saved to the disk.
1. Be familiar with the usage of doris' import function and internal implementation process
2. Add a new be configuration item load_error_log_limit_bytes = default value 200MB
3. Use the newly added threshold to limit the amount of data that RuntimeState::append_error_msg_to_file writes to disk
4. Write regression cases for testing and verification
---
Description in Chinese:
在导入过程中，如果原始数据有问题，那么我们会把错误数据存放到磁盘上的一个 error_log 文件中方便后续 debug。但是如果错误数据很多，就会占用大量的磁盘空间。所以需要限制落盘的错误数据数量。
1. 熟悉 doris 的导入功能用法和内部实现流程
2. 增加新的 be 配置项目 load_error_log_limit_bytes = 默认值 200MB
3. 使用新增的阈值限制 RuntimeState::append_error_msg_to_file 落盘数据量
4. 编写回归case进行测试和验证
### Solution
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Enhancement] limit the amount of error_log when load to save disks [doris]
Posted by "xy720 (via GitHub)" <gi...@apache.org>.
xy720 closed issue #27481: [Enhancement] limit the amount of error_log when load to save disks
URL: https://github.com/apache/doris/issues/27481
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Enhancement] limit the amount of error_log when load to save disks [doris]
Posted by "freemandealer (via GitHub)" <gi...@apache.org>.
freemandealer commented on issue #27481:
URL: https://github.com/apache/doris/issues/27481#issuecomment-1824081405
> felixwluo
Nice! Don't hesitate to contact me when you need help.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Enhancement] limit the amount of error_log when load to save disks [doris]
Posted by "felixwluo (via GitHub)" <gi...@apache.org>.
felixwluo commented on issue #27481:
URL: https://github.com/apache/doris/issues/27481#issuecomment-1824097611
please assigned
> > felixwluo
>
> Nice! Don't hesitate to contact me when you need help.
Okay，Please assign it to me. Thank you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Enhancement] limit the amount of error_log when load to save disks [doris]
Posted by "felixwluo (via GitHub)" <gi...@apache.org>.
felixwluo commented on issue #27481:
URL: https://github.com/apache/doris/issues/27481#issuecomment-1824078966
I will pick
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org