You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@rocketmq.apache.org by "1925747950 (via GitHub)" <gi...@apache.org> on 2024/01/10 10:13:49 UTC
[I] 磁盘空间充足却频繁告警 [rocketmq]
1925747950 opened a new issue, #7738:
URL: https://github.com/apache/rocketmq/issues/7738
### Before Creating the Bug Report
- [X] I found a bug, not just asking a question, which should be created in [GitHub Discussions](https://github.com/apache/rocketmq/discussions).
- [X] I have searched the [GitHub Issues](https://github.com/apache/rocketmq/issues) and [GitHub Discussions](https://github.com/apache/rocketmq/discussions) of this repository and believe that this is not a duplicate.
- [X] I have confirmed that this bug belongs to the current repository, not other repositories of RocketMQ.
### Runtime platform environment
$ sudo cat /etc/os-release
NAME="Anolis OS"
VERSION="8.8"
ID="anolis"
ID_LIKE="rhel fedora centos"
VERSION_ID="8.8"
PLATFORM_ID="platform:an8"
PRETTY_NAME="Anolis OS 8.8"
ANSI_COLOR="0;31"
HOME_URL="https://openanolis.cn/"
### RocketMQ version
rocketmq-all-5.1.3-bin-release
### JDK Version
$ java -version
java version "1.8.0_202"
Java(TM) SE Runtime Environment (build 1.8.0_202-b08)
Java HotSpot(TM) 64-Bit Server VM (build 25.202-b08, mixed mode)
### Describe the Bug
目前mq生产业务3台虚机部署的三主三从集群，所使用的磁盘data总容量为500G，总会在几天内磁盘使用率从4%上升到100%触发告警，达到100%后立即恢复正常使用率4%，三台虚机仅部署rocketmq，且均出现此现象，同一局域网内其他虚机一切正常，且都是使用同一模板创建虚机，无法定位此现象原因，，，特来寻求帮助，有人遇到过这个现象吗？
### Steps to Reproduce
生产环境正式启用后出现
### What Did You Expect to See?
希望能得到解决办法
### What Did You See Instead?
emmm
### Additional Context
nil
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Insufficient disk space but frequent alarms [rocketmq]
Posted by "1925747950 (via GitHub)" <gi...@apache.org>.
1925747950 commented on issue #7738:
URL: https://github.com/apache/rocketmq/issues/7738#issuecomment-1887345872
> 可以看一下
>
> 1. 这个监控中的指标，是怎么获取的
> 2. 在上涨期间，可以观察一下第一个那个指标在机器上的那个对应的信息。
监控指标是基础磁盘指标，告警时登录服务器使用df命令查看/data占用接近500G，使用率也接近100%，但是使用du命令查看又只占用20G左右，等待磁盘达到100%后就立刻恢复成正常的4%使用率。。。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Insufficient disk space but frequent alarms [rocketmq]
Posted by "leizhiyuan (via GitHub)" <gi...@apache.org>.
leizhiyuan commented on issue #7738:
URL: https://github.com/apache/rocketmq/issues/7738#issuecomment-1891548229
核心说明你的这些日志打印太多了。日志占用的很多，日志删除是log组件滚动删除的，滚动到otherday目录
可以看看日志里面都是啥日志内容多，默认INFO应该没多少日志。除非业务上有一些什么特殊的，导致一致在打印某些日志。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Insufficient disk space but frequent alarms [rocketmq]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #7738: Insufficient disk space but frequent alarms
URL: https://github.com/apache/rocketmq/issues/7738
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Insufficient disk space but frequent alarms [rocketmq]
Posted by "leizhiyuan (via GitHub)" <gi...@apache.org>.
leizhiyuan commented on issue #7738:
URL: https://github.com/apache/rocketmq/issues/7738#issuecomment-1920622900
any update ? @1925747950
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Insufficient disk space but frequent alarms [rocketmq]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #7738:
URL: https://github.com/apache/rocketmq/issues/7738#issuecomment-2628599957
This issue is stale because it has been open for 365 days with no activity. It will be closed in 3 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Insufficient disk space but frequent alarms [rocketmq]
Posted by "leizhiyuan (via GitHub)" <gi...@apache.org>.
leizhiyuan commented on issue #7738:
URL: https://github.com/apache/rocketmq/issues/7738#issuecomment-1888143893
df和du统计对于已删除句柄但是还没有真正删除的有差异。当df统计到很大的时候。可以
lsof -n|grep deleted 看一下，参考
https://www.cnblogs.com/heyonggang/p/3644736.html
HGQ ***@***.***> 于 2024年1月11日周四 下午10:50写道：
> 可以看一下
>
> 1. 这个监控中的指标，是怎么获取的
> 2. 在上涨期间，可以观察一下第一个那个指标在机器上的那个对应的信息。
>
>
> 监控指标是基础磁盘指标，告警时登录服务器使用df命令查看/data占用接近500G，使用率也接近100%，但是使用du命令查看又只占用20G左右，等待磁盘达到100%后就立刻恢复成正常的4%使用率。。。
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/rocketmq/issues/7738#issuecomment-1887345872>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAUPLYBFB63KJGPLGM6DQP3YN733FAVCNFSM6AAAAABBUQS3T6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQOBXGM2DKOBXGI>
> .
> You are receiving this because you commented.Message ID:
> ***@***.***>
>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Insufficient disk space but frequent alarms [rocketmq]
Posted by "RongtongJin (via GitHub)" <gi...@apache.org>.
RongtongJin commented on issue #7738:
URL: https://github.com/apache/rocketmq/issues/7738#issuecomment-1884612263
出现问题的时候看看哪个文件占用高
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Insufficient disk space but frequent alarms [rocketmq]
Posted by "leizhiyuan (via GitHub)" <gi...@apache.org>.
leizhiyuan commented on issue #7738:
URL: https://github.com/apache/rocketmq/issues/7738#issuecomment-1886948508
可以看一下
1. 这个监控中的指标，是怎么获取的
2. 在上涨期间，可以观察一下第一个那个指标在机器上的那个对应的信息。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Insufficient disk space but frequent alarms [rocketmq]
Posted by "leizhiyuan (via GitHub)" <gi...@apache.org>.
leizhiyuan commented on issue #7738:
URL: https://github.com/apache/rocketmq/issues/7738#issuecomment-1891561489
我好奇，你这个共用，多线程使用同一个文件， 比如图上的remoting.log，感觉会有问题呀，你不同的进程，也可以设置个不同的子目录再看看吧
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Insufficient disk space but frequent alarms [rocketmq]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #7738:
URL: https://github.com/apache/rocketmq/issues/7738#issuecomment-2635414445
This issue was closed because it has been inactive for 3 days since being marked as stale.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Insufficient disk space but frequent alarms [rocketmq]
Posted by "1925747950 (via GitHub)" <gi...@apache.org>.
1925747950 commented on issue #7738:
URL: https://github.com/apache/rocketmq/issues/7738#issuecomment-1891525319
> df和du统计对于已删除句柄但是还没有真正删除的有差异。当df统计到很大的时候。可以 lsof -n|grep deleted 看一下，参考 https://www.cnblogs.com/heyonggang/p/3644736.html HGQ ***@***.***> 于 2024年1月11日周四 下午10:50写道：
> […](#)
> 可以看一下 1. 这个监控中的指标，是怎么获取的 2. 在上涨期间，可以观察一下第一个那个指标在机器上的那个对应的信息。 监控指标是基础磁盘指标，告警时登录服务器使用df命令查看/data占用接近500G，使用率也接近100%，但是使用du命令查看又只占用20G左右，等待磁盘达到100%后就立刻恢复成正常的4%使用率。。。 — Reply to this email directly, view it on GitHub <[#7738 (comment)](https://github.com/apache/rocketmq/issues/7738#issuecomment-1887345872)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAUPLYBFB63KJGPLGM6DQP3YN733FAVCNFSM6AAAAABBUQS3T6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQOBXGM2DKOBXGI> . You are receiving this because you commented.Message ID: ***@***.***>
感谢，使用lsof能找到占用比较大的文件：/data/logs/rocketmqlogs/remoting.log (deleted)，但是为什么会是delete状态呢，是mq自动进行的删除吗？（是否与集群的两个实例写同一个日志文件有关系？）
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Insufficient disk space but frequent alarms [rocketmq]
Posted by "lizhimins (via GitHub)" <gi...@apache.org>.
lizhimins commented on issue #7738:
URL: https://github.com/apache/rocketmq/issues/7738#issuecomment-1899618374
磁盘使用率这个描述很奇怪啊，是 ioutil 还是 disk usage
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Insufficient disk space but frequent alarms [rocketmq]
Posted by "1925747950 (via GitHub)" <gi...@apache.org>.
1925747950 commented on issue #7738:
URL: https://github.com/apache/rocketmq/issues/7738#issuecomment-1886378673
> 出现问题的时候看看哪个文件占用高
文件占用都不高：使用du -sh /data/*查看目录占用只用20G左右，并且没有看到大的隐藏文件
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org