You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "sai-krystal (via GitHub)" <gi...@apache.org> on 2023/12/06 02:11:44 UTC
[I] [Bug] BE在数据均衡时会将冗余数据放入回收站中(catalog_trash_expire_second默认一天),导致当前BE节点DataUsedCapacity不高的情况下一直将数据迁移至其他BE节点 [doris]
sai-krystal opened a new issue, #28038:
URL: https://github.com/apache/doris/issues/28038
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues.
### Version
2.0.2
### What's Wrong?
BE在数据均衡时会将冗余数据放入回收站中(catalog_trash_expire_second默认一天),导致当前BE节点DataUsedCapacity不高的情况下一直将数据迁移至其他BE节点
### What You Expected?
BE数据均衡时冗余数据不放入回收站,因为不是用户自己删除的数据。
### How to Reproduce?
_No response_
### Anything Else?
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Bug] BE在数据均衡时会将冗余数据放入回收站中(catalog_trash_expire_second默认一天),导致当前BE节点DataUsedCapacity不高的情况下一直将数据迁移至其他BE节点。当前BE节点UsedPct降不下来(因为回收站冗余数据需要等待一天才会自动释放)。 [doris]
Posted by "sai-krystal (via GitHub)" <gi...@apache.org>.
sai-krystal commented on issue #28038:
URL: https://github.com/apache/doris/issues/28038#issuecomment-1845324850
> catalog recycle bin 中的数据过了一天之后超时了,它们才会被删除,接着挪到trash中。此时也无法通过recover 的手段去会恢复这部分数据。成为trash之后,这部分数据基本也费了,很难救可以。可以直接早点淘汰
--------------------------------------------------------------------------------------
明白了,谢谢解答
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Bug] BE在数据均衡时会将冗余数据放入回收站中(catalog_trash_expire_second默认一天),导致当前BE节点DataUsedCapacity不高的情况下一直将数据迁移至其他BE节点。当前BE节点UsedPct降不下来(因为回收站冗余数据需要等待一天才会自动释放)。 [doris]
Posted by "sai-krystal (via GitHub)" <gi...@apache.org>.
sai-krystal commented on issue #28038:
URL: https://github.com/apache/doris/issues/28038#issuecomment-1843960142

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Bug] BE在数据均衡时会将冗余数据放入回收站中(catalog_trash_expire_second默认一天),导致当前BE节点DataUsedCapacity不高的情况下一直将数据迁移至其他BE节点。当前BE节点UsedPct降不下来(因为回收站冗余数据需要等待一天才会自动释放)。 [doris]
Posted by "sai-krystal (via GitHub)" <gi...@apache.org>.
sai-krystal closed issue #28038: [Bug] BE在数据均衡时会将冗余数据放入回收站中(catalog_trash_expire_second默认一天),导致当前BE节点DataUsedCapacity不高的情况下一直将数据迁移至其他BE节点。当前BE节点UsedPct降不下来(因为回收站冗余数据需要等待一天才会自动释放)。
URL: https://github.com/apache/doris/issues/28038
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Bug] BE在数据均衡时会将冗余数据放入回收站中(catalog_trash_expire_second默认一天),导致当前BE节点DataUsedCapacity不高的情况下一直将数据迁移至其他BE节点。当前BE节点UsedPct降不下来(因为回收站冗余数据需要等待一天才会自动释放)。 [doris]
Posted by "sai-krystal (via GitHub)" <gi...@apache.org>.
sai-krystal commented on issue #28038:
URL: https://github.com/apache/doris/issues/28038#issuecomment-1845134890
> 均衡时,不衡量DataUasedCapacity衡量,是以实际磁盘使用率UsedPct来区分。用户手工admin clean trash来清除垃圾。可以降低be.conf 的trash_file_expire_time_sec (默认三天)将为1小时,把trash 早点自动淘汰
用户手动删除数据跟BE均衡的数据要区别对待。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Bug] BE在数据均衡时会将冗余数据放入回收站中(catalog_trash_expire_second默认一天),导致当前BE节点DataUsedCapacity不高的情况下一直将数据迁移至其他BE节点。当前BE节点UsedPct降不下来(因为回收站冗余数据需要等待一天才会自动释放)。 [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on issue #28038:
URL: https://github.com/apache/doris/issues/28038#issuecomment-1845211332
catalog recycle bin 中的数据过了一天之后超时了,它们才会被删除,接着挪到trash中。此时也无法通过recover 的手段去会恢复这部分数据。成为trash之后,这部分数据基本也费了,很难救可以。可以直接早点淘汰
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Bug] BE在数据均衡时会将冗余数据放入回收站中(catalog_trash_expire_second默认一天),导致当前BE节点DataUsedCapacity不高的情况下一直将数据迁移至其他BE节点。当前BE节点UsedPct降不下来(因为回收站冗余数据需要等待一天才会自动释放)。 [doris]
Posted by "sai-krystal (via GitHub)" <gi...@apache.org>.
sai-krystal commented on issue #28038:
URL: https://github.com/apache/doris/issues/28038#issuecomment-1845141701
> 均衡时,不衡量DataUasedCapacity衡量,是以实际磁盘使用率UsedPct来区分。用户手工admin clean trash来清除垃圾。可以降低be.conf 的trash_file_expire_time_sec (默认三天)将为1小时,把trash 早点自动淘汰
-----------------------------------------------------------------------
用户手动删除数据跟BE均衡的数据需要区别对待。如果修改be.conf 的trash_file_expire_time_sec的配置改为1小时,那么用户手动失误删除的数据容易来不及恢复。而BE均衡的数据是用户无感知的,但却占用了磁盘容量。

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Bug] BE在数据均衡时会将冗余数据放入回收站中(catalog_trash_expire_second默认一天),导致当前BE节点DataUsedCapacity不高的情况下一直将数据迁移至其他BE节点。当前BE节点UsedPct降不下来(因为回收站冗余数据需要等待一天才会自动释放)。 [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on issue #28038:
URL: https://github.com/apache/doris/issues/28038#issuecomment-1844269468
均衡时,不衡量DataUasedCapacity衡量,是以实际磁盘使用率UsedPct来区分。用户手工admin clean trash来清除垃圾。可以设be.conf 的trash_file_expire_time_sec(默认三天)把trash 早点淘汰
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Bug] BE在数据均衡时会将冗余数据放入回收站中(catalog_trash_expire_second默认一天),导致当前BE节点DataUsedCapacity不高的情况下一直将数据迁移至其他BE节点。当前BE节点UsedPct降不下来(因为回收站冗余数据需要等待一天才会自动释放)。 [doris]
Posted by "sai-krystal (via GitHub)" <gi...@apache.org>.
sai-krystal commented on issue #28038:
URL: https://github.com/apache/doris/issues/28038#issuecomment-1845342532
> catalog recycle bin 中的数据过了一天之后超时了,它们才会被删除,接着挪到trash中。此时也无法通过recover 的手段去会恢复这部分数据。成为trash之后,这部分数据基本也费了,很难救可以。可以直接早点淘汰
---------------------------------------------------------------------------------------------------------------------
请问下show catalog recycle bin查询出来的数据,用什么方式可以不用等待一天。在不调整配置的情况下手动释放空间?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Bug] BE在数据均衡时会将冗余数据放入回收站中(catalog_trash_expire_second默认一天),导致当前BE节点DataUsedCapacity不高的情况下一直将数据迁移至其他BE节点。当前BE节点UsedPct降不下来(因为回收站冗余数据需要等待一天才会自动释放)。 [doris]
Posted by "sai-krystal (via GitHub)" <gi...@apache.org>.
sai-krystal commented on issue #28038:
URL: https://github.com/apache/doris/issues/28038#issuecomment-1845163456
> 均衡时,不衡量DataUasedCapacity衡量,是以实际磁盘使用率UsedPct来区分。用户手工admin clean trash来清除垃圾。可以降低be.conf 的trash_file_expire_time_sec (默认三天)将为1小时,把trash 早点自动淘汰
---------------------------------------------------------------------------------------------------
我知道该如何解决这种情况,我只是发现了Doris集群在数据均衡时出了这种问题。以及给出“BE数据均衡时冗余数据不放入回收站,因为不是用户自己删除的数据。”这种建议。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Bug] BE在数据均衡时会将冗余数据放入回收站中(catalog_trash_expire_second默认一天),导致当前BE节点DataUsedCapacity不高的情况下一直将数据迁移至其他BE节点。当前BE节点UsedPct降不下来(因为回收站冗余数据需要等待一天才会自动释放)。 [doris]
Posted by "sai-krystal (via GitHub)" <gi...@apache.org>.
sai-krystal commented on issue #28038:
URL: https://github.com/apache/doris/issues/28038#issuecomment-1845159216
> 均衡时,不衡量DataUasedCapacity衡量,是以实际磁盘使用率UsedPct来区分。用户手工admin clean trash来清除垃圾。可以降低be.conf 的trash_file_expire_time_sec (默认三天)将为1小时,把trash 早点自动淘汰
----------------------------------------------------------------------------------------------------
自动化均衡数据时,应该无需用户手动admin clean trash进行垃圾清理的操作,而是系统自动进行处理,用户无感知。缩短回收站的时间配置可能会导致数据在短时间内被删除而无法及时恢复。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Bug] BE在数据均衡时会将冗余数据放入回收站中(catalog_trash_expire_second默认一天),导致当前BE节点DataUsedCapacity不高的情况下一直将数据迁移至其他BE节点。当前BE节点UsedPct降不下来(因为回收站冗余数据需要等待一天才会自动释放)。 [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on issue #28038:
URL: https://github.com/apache/doris/issues/28038#issuecomment-1845203004
处在 catalog recycle bin 中的数据不会被放入到trash中, 它们依旧是在data目录下的。BE 端的trash 才是实际费掉的数据, 它主要是为了防止代码出bug不小心全删数据后,通过人工干预的方式尝试最后挽救一下数据。但随着doris这么多年的修改,BE的trash 用处不大,处在BE trash中的数据,它们的tablet 元数据已经没了, 要在实际中想通过它们捞回数据是很麻烦的。
所以,说BE trash用处现在不大,也基本没用户会通过它来捞回数据, 直接把BE的trash超时时间调小就ok了
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org