You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hugegraph.apache.org by "wuchaojing (via GitHub)" <gi...@apache.org> on 2023/04/06 10:18:22 UTC
[GitHub] [incubator-hugegraph-toolchain] wuchaojing opened a new issue, #449: [Bug] 边shard不均导致backup很慢
wuchaojing opened a new issue, #449:
URL: https://github.com/apache/incubator-hugegraph-toolchain/issues/449
### Bug Type (问题类型)
performance (性能下降)
### Before submit
- [X] 我已经确认现有的 [Issues](https://github.com/apache/hugegraph/issues) 与 [FAQ](https://hugegraph.apache.org/docs/guides/faq/) 中没有相同 / 重复问题 (I have confirmed and searched that there are no similar problems in the historical issue and documents)
### Environment (环境信息)
- Server Version: 1.0.0 (Apache Release Version)
- Backend: RocksDB x nodes, HDD or SSD
- OS: xx CPUs, xx G RAM, Ubuntu 2x.x / CentOS 7.x
- Data Size: xx vertices, xx edges <!-- (like 1000W 点, 9000W 边) -->
### Expected & Actual behavior (期望与实际表现)
执行边的备份,发现非常慢(点备份很快),命令为:
`bin/hugegraph --url http://127.0.0.1:8080 --graph hugegraph backup -t edge --directory ./backup_test`
通过源码发现问题应该是边shard不均匀,导致大部分的线程执行完各自的小分片之后退出,最后只会有1~2个线程在执行较大的分片。虽然配置了多线程,但最后只有一个线程在执行备份,其他线程均退出。
<img width="504" alt="image" src="https://user-images.githubusercontent.com/37741352/230309159-ae1229f4-a3ee-4916-8d54-e8f671d53126.png">
也打印了线程池的活跃情况,一开始全部活跃,到最后只有2个活跃。(忽略线程数与备份生成的文件数不一致的情况,修改过线程数)
<img width="1186" alt="image" src="https://user-images.githubusercontent.com/37741352/230309830-85661d90-4d4c-4936-9e42-26fe3fa24fc5.png">
<img width="1083" alt="image" src="https://user-images.githubusercontent.com/37741352/230309512-4c330b8c-b726-4110-b5de-b866d8e1ad50.png">
### Vertex/Edge example (问题点 / 边数据举例)
_No response_
### Schema [VertexLabel, EdgeLabel, IndexLabel] (元数据结构)
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@hugegraph.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-hugegraph-toolchain] zyxxoo commented on issue #449: [Bug] 边shard不均导致backup很慢
Posted by "zyxxoo (via GitHub)" <gi...@apache.org>.
zyxxoo commented on issue #449:
URL: https://github.com/apache/incubator-hugegraph-toolchain/issues/449#issuecomment-1499992236
是什么后端呢?rocksdb 吗?现在的实现确实会有不均衡的问题
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
[GitHub] [incubator-hugegraph-toolchain] wuchaojing commented on issue #449: [Bug] 边shard不均导致backup很慢
Posted by "wuchaojing (via GitHub)" <gi...@apache.org>.
wuchaojing commented on issue #449:
URL: https://github.com/apache/incubator-hugegraph-toolchain/issues/449#issuecomment-1499886128
请问下这个问题有解决的思路吗,是需要优化server的splits接口,还是需要优化tools的backup方法呢?如果不复杂的话,我可以尝试解决一下~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
[GitHub] [incubator-hugegraph-toolchain] imbajin commented on issue #449: [Bug] 边shard不均导致backup很慢
Posted by "imbajin (via GitHub)" <gi...@apache.org>.
imbajin commented on issue #449:
URL: https://github.com/apache/incubator-hugegraph-toolchain/issues/449#issuecomment-1498840962
Thanks for the feedback.
简单来说, 因为有很一段时间 tools 原本的 owner / 作者暂时没有在社区持续更新它, 所以是存在一些 gap, 我们会尽快在 4~5 月找下**总体**改进的方式, 包括 `hubble` / `loader` 也会有很大的更新/扩展, 欢迎关注和反馈现存的问题. 如果愿意一起参与改进, 也欢迎联系我们保持进度条的更新~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
[GitHub] [incubator-hugegraph-toolchain] imbajin commented on issue #449: [Bug] 边shard不均导致backup很慢
Posted by "imbajin (via GitHub)" <gi...@apache.org>.
imbajin commented on issue #449:
URL: https://github.com/apache/incubator-hugegraph-toolchain/issues/449#issuecomment-1499889270
> 请问下这个问题有解决的思路吗,是需要优化 server 的 splits 接口,还是需要优化 tools 的 backup 方法呢?如果不复杂的话,我可以尝试解决一下~
我理解是都需要优化, 不过 server 这边肯定麻烦一些, tools 这边之后我看看改进后的版本情况反馈一下, 包括整体思路
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
[GitHub] [incubator-hugegraph-toolchain] wuchaojing commented on issue #449: [Bug] 边shard不均导致backup很慢
Posted by "wuchaojing (via GitHub)" <gi...@apache.org>.
wuchaojing commented on issue #449:
URL: https://github.com/apache/incubator-hugegraph-toolchain/issues/449#issuecomment-1500037556
> 是什么后端呢?rocksdb 吗?现在的实现确实会有不均衡的问题
是的,rocksdb后端
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
[GitHub] [incubator-hugegraph-toolchain] wuchaojing commented on issue #449: [Bug] 边shard不均导致backup很慢
Posted by "wuchaojing (via GitHub)" <gi...@apache.org>.
wuchaojing commented on issue #449:
URL: https://github.com/apache/incubator-hugegraph-toolchain/issues/449#issuecomment-1498967285
感谢回复,期待更新ing😄
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org