You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "conanceph (via GitHub)" <gi...@apache.org> on 2025/12/15 10:30:16 UTC
[I] [Bug] doris be 数据占用磁盘的容量，要比实际数据占用的多得多，且没办法释放 [doris]
conanceph opened a new issue, #59047:
URL: https://github.com/apache/doris/issues/59047
### Search before asking
- [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues.
### Version
doris version：3.1.0
### What's Wrong?
背景介绍：
1. 使用 k8s 部署 doris，be是3实例，并且配置使用 ceph 做为后端 pvc 卷的实际存储设备
2. 具体环境信息、异常内容如下：
``` shell
[root@dev-int-pf-01 ~]# kubectl get pod | grep doriscluster-helm-be
doriscluster-helm-be-0 1/1 Running 0 49m
doriscluster-helm-be-1 1/1 Running 0 50m
doriscluster-helm-be-2 1/1 Running 0 50m
[root@dev-int-pf-01 ~]#
[root@dev-int-hkpf-01 ~]# kubectl get pvc | grep be-storage
be-storage-doriscluster-helm-be-0 Bound pvc-12028aef-ba54-491a-a875-1a6203d40fd1 100Gi RWO rook-ceph-block <unset> 11d
be-storage-doriscluster-helm-be-1 Bound pvc-0f8bd60c-a173-4a8b-893d-288885c250f5 100Gi RWO rook-ceph-block <unset> 11d
be-storage-doriscluster-helm-be-2 Bound pvc-daeedfa7-3b6d-4343-bf73-972dc71dba01 100Gi RWO rook-ceph-block <unset> 11d
[root@dev-int-hkpf-01 ~]#
[root@dev-int-hkpf-01 ~]# kubectl get pv | grep be-storage
pvc-0f8bd60c-a173-4a8b-893d-288885c250f5 100Gi RWO Delete Bound default/be-storage-doriscluster-helm-be-1 rook-ceph-block <unset> 11d
pvc-12028aef-ba54-491a-a875-1a6203d40fd1 100Gi RWO Delete Bound default/be-storage-doriscluster-helm-be-0 rook-ceph-block <unset> 11d
pvc-daeedfa7-3b6d-4343-bf73-972dc71dba01 100Gi RWO Delete Bound default/be-storage-doriscluster-helm-be-2 rook-ceph-block <unset> 11d
# 拿 be-storage-doriscluster-helm-be-0 这个 pv 举例：
[root@dev-int-hkpf-01 ~]# kubectl describe pv pvc-12028aef-ba54-491a-a875-1a6203d40fd1 | grep imageName
imageName=csi-vol-1a89d830-ea89-4ec9-bb23-96569997bab0
[root@dev-int-hkpf-01 ~]#
# 可以看到，pv 对应的 ceph rbd 设备已经被占用了85G
[root@rook-ceph-tools-64d98bcb68-tmxc7 /]# rbd du replicapool/csi-vol-1a89d830-ea89-4ec9-bb23-96569997bab0
warning: fast-diff map is not enabled for csi-vol-1a89d830-ea89-4ec9-bb23-96569997bab0. operation may be slow.
NAME PROVISIONED USED
csi-vol-1a89d830-ea89-4ec9-bb23-96569997bab0 100 GiB 85 GiB
[root@rook-ceph-tools-64d98bcb68-tmxc7 /]#
# 但在 dori-be-0 pod实例中看到只使用了8G
root@doriscluster-helm-be-0:/opt/apache-doris# du -h --max-depth=1 /opt/apache-doris/be
24K /opt/apache-doris/be/bin
24K /opt/apache-doris/be/conf
21M /opt/apache-doris/be/dict
5.2G /opt/apache-doris/be/lib
620K /opt/apache-doris/be/licenses
1.4G /opt/apache-doris/be/log
0 /opt/apache-doris/be/plugins
8.0G /opt/apache-doris/be/storage
60K /opt/apache-doris/be/tools
2.6M /opt/apache-doris/be/www
15G /opt/apache-doris/be
root@doriscluster-helm-be-0:/opt/apache-doris#
root@doriscluster-helm-be-0:/opt/apache-doris# df -Th /opt/apache-doris/be/storage
Filesystem Type Size Used Avail Use% Mounted on
/dev/rbd0 ext4 98G 8.0G 90G 9% /opt/apache-doris/be/storage
root@doriscluster-helm-be-0:/opt/apache-doris#
```
3. 尝试过的处理方法：
参考 issue 修改了如下参数：
https://github.com/apache/doris/issues/30016
https://github.com/apache/doris/issues/31501
``` shell
# 修改 doris-be-configmap 的如下参数，该参数用于控制垃圾回收（GC）检查线程的运行间隔。定义了 Doris 后台的垃圾回收检查线程多久被唤醒一次，以扫描并清理不再需要的垃圾数据文件（例如，因数据合并、删除或导入失败产生的残留文件）
path_gc_check_interval_second = 120
```
但发现并不管用，同时也重启了 doris-be ，还是一样的没有释放ceph存储容量
4. 检查过 ceph 本身并没有对这些 rbd image 做快照，并且其他服务在删除后，容量是能够正常释放的
### What You Expected?
doris be 或 fe 实际的数据大小，应该与占用的实际磁盘大小一致
### How to Reproduce?
_No response_
### Anything Else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Bug] doris be 数据占用磁盘的容量，要比实际数据占用的多得多，且没办法释放 [doris]
Posted by "conanceph (via GitHub)" <gi...@apache.org>.
conanceph closed issue #59047: [Bug] doris be 数据占用磁盘的容量，要比实际数据占用的多得多，且没办法释放
URL: https://github.com/apache/doris/issues/59047
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Bug] doris be 数据占用磁盘的容量，要比实际数据占用的多得多，且没办法释放 [doris]
Posted by "conanceph (via GitHub)" <gi...@apache.org>.
conanceph commented on issue #59047:
URL: https://github.com/apache/doris/issues/59047#issuecomment-3655369070
解决方案：
我使用的机器数据盘都是SSD，需在所有ceph机器上执行 fstrim -av后即可，理由如下：
HDD核心差异：
HDD 采用磁性盘片存储数据，允许直接覆盖写入。当您在文件系统中删除文件时，HDD的物理空间会立即被标记为可用，新数据可以直接写入这些空间，无需先擦除再写入。因此，HDD不会出现"逻辑空间已释放但物理空间仍被占用"的情况。
SSD 则完全不同。SSD采用闪存芯片存储数据，遵循"先擦除再写入"的物理规则。当文件被删除时，操作系统只是将数据标记为"可重用"，但SSD底层并不知道这些数据已经无效。只有执行fstrim命令发送TRIM指令后，SSD才会真正擦除这些数据块，释放物理空间。
为什么需要fstrim：
fstrim命令是专门为SSD设计的，用于通知SSD控制器哪些数据块已经不再使用。如果不执行fstrim，SSD在进行垃圾回收时仍然会移动那些已删除但未标记的数据，导致写入性能下降、闪存寿命缩短。
在HDD上部署Ceph RBD的场景：
如果在HDD上部署Ceph RBD服务，不会遇到文件删除后空间不释放的问题。因为HDD允许直接覆盖写入，删除文件后物理空间会立即被标记为可用，无需TRIM机制。这也是为什么在HDD上执行fstrim命令通常不会有任何效果的原因。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org