You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@inlong.apache.org by GitBox <gi...@apache.org> on 2022/11/15 02:20:17 UTC
[GitHub] [inlong] FvvSuu opened a new issue, #6535: [Improve][TubeMQ]The storage space used by the offset of TubeMQ broker is too large
FvvSuu opened a new issue, #6535:
URL: https://github.com/apache/inlong/issues/6535
### Description
The storage space used by the offset of TubeMQ broker is too large.
<img width="725" alt="image" src="https://user-images.githubusercontent.com/53012896/201810458-3d0336d6-4e91-41a6-ad42-200600884369.png">
There are 5~6 consumer groups in the picture
### InLong Component
InLong TubeMQ
### Are you willing to submit PR?
- [ ] Yes, I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@inlong.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] gosonzhang closed issue #6535: [Improve][TubeMQ]The storage space used by the offset of TubeMQ broker is too large
Posted by GitBox <gi...@apache.org>.
gosonzhang closed issue #6535: [Improve][TubeMQ]The storage space used by the offset of TubeMQ broker is too large
URL: https://github.com/apache/inlong/issues/6535
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] gosonzhang commented on issue #6535: [Improve][TubeMQ]The storage space used by the offset of TubeMQ broker is too large
Posted by GitBox <gi...@apache.org>.
gosonzhang commented on issue #6535:
URL: https://github.com/apache/inlong/issues/6535#issuecomment-1314705740
It should be caused by too much duplicate data in the offset records of the storage consumption group: in order to make the records more readable[1], for example, each partition consumption record contains the consumption group name, and each partition repeatedly records the maximum and minimum offsets, as a result, the total data record is bloated too much.
Considering the traversal of the text display, I plan to still use the text to represent the saved historical offset, but perform field key abbreviation and deduplication processing to reduce the length of the total record.
1. https://github.com/apache/inlong/blob/master/inlong-tubemq/tubemq-server/src/main/java/org/apache/inlong/tubemq/server/broker/offset/OffsetRecordInfo.java#L68
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] gosonzhang commented on issue #6535: [Improve][TubeMQ]The storage space used by the offset of TubeMQ broker is too large
Posted by GitBox <gi...@apache.org>.
gosonzhang commented on issue #6535:
URL: https://github.com/apache/inlong/issues/6535#issuecomment-1314670024
Thanks @FvvSuu, got it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org