You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by GitBox <gi...@apache.org> on 2022/02/11 07:57:59 UTC
[GitHub] [skywalking-banyandb] hanahmily opened a new pull request #78: Extend tsdb to contain streams of a group
hanahmily opened a new pull request #78:
URL: https://github.com/apache/skywalking-banyandb/pull/78
A stream would create a tsdb, which causes the banyandb to occupy too much memory.
This tends to extend a single tsdb's capacity to store all streams in a single group. These changes are:
* Reuse the tsdb in the stream module.
* Add Group resource to set the resource options, like shard number, interval rules, etc.
* Update query, liaison, and other relevant modules.
* Remove periodic reconciliation between stream and metadata. We should enhance the notification framework of metadata changing to relay the events to stream and other modules.
* Remove the measuring module. Given that the changes are too many, we will add them back in following-up commits
@lujiajing1126 Please verify whether this could reduce memory consumption on integrating with the OAP.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-banyandb] lujiajing1126 edited a comment on pull request #78: Extend tsdb to contain streams of a group
Posted by GitBox <gi...@apache.org>.
lujiajing1126 edited a comment on pull request #78:
URL: https://github.com/apache/skywalking-banyandb/pull/78#issuecomment-1038150088
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-banyandb] lujiajing1126 merged pull request #78: Extend tsdb to contain streams of a group
Posted by GitBox <gi...@apache.org>.
lujiajing1126 merged pull request #78:
URL: https://github.com/apache/skywalking-banyandb/pull/78
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-banyandb] lujiajing1126 edited a comment on pull request #78: Extend tsdb to contain streams of a group
Posted by GitBox <gi...@apache.org>.
lujiajing1126 edited a comment on pull request #78:
URL: https://github.com/apache/skywalking-banyandb/pull/78#issuecomment-1038150088
I confirm that this branch can significantly reduce the memory usage (from >10 GB to about a few hundreds MB). But some issues,
<img width="1330" alt="image" src="https://user-images.githubusercontent.com/2568208/153757321-9c0d16ef-8d1a-4aaf-8fcf-7b7ca4ffe5cc.png">
@hanahmily then we should merge this PR first or handle these known issues?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-banyandb] lujiajing1126 edited a comment on pull request #78: Extend tsdb to contain streams of a group
Posted by GitBox <gi...@apache.org>.
lujiajing1126 edited a comment on pull request #78:
URL: https://github.com/apache/skywalking-banyandb/pull/78#issuecomment-1038150088
I confirm that this branch significantly reduce the memory usage from (>10 GB to about a few hundreds MB). But some issues,
<img width="1330" alt="image" src="https://user-images.githubusercontent.com/2568208/153757321-9c0d16ef-8d1a-4aaf-8fcf-7b7ca4ffe5cc.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-banyandb] lujiajing1126 commented on pull request #78: Extend tsdb to contain streams of a group
Posted by GitBox <gi...@apache.org>.
lujiajing1126 commented on pull request #78:
URL: https://github.com/apache/skywalking-banyandb/pull/78#issuecomment-1035989788
I will make a try with this PR and give some feedback
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-banyandb] lujiajing1126 edited a comment on pull request #78: Extend tsdb to contain streams of a group
Posted by GitBox <gi...@apache.org>.
lujiajing1126 edited a comment on pull request #78:
URL: https://github.com/apache/skywalking-banyandb/pull/78#issuecomment-1038150088
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-banyandb] lujiajing1126 commented on pull request #78: Extend tsdb to contain streams of a group
Posted by GitBox <gi...@apache.org>.
lujiajing1126 commented on pull request #78:
URL: https://github.com/apache/skywalking-banyandb/pull/78#issuecomment-1038150088
I confirm that this branch significantly reduce the memory usage. But some issues,
<img width="1330" alt="image" src="https://user-images.githubusercontent.com/2568208/153757321-9c0d16ef-8d1a-4aaf-8fcf-7b7ca4ffe5cc.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-banyandb] lujiajing1126 commented on pull request #78: Extend tsdb to contain streams of a group
Posted by GitBox <gi...@apache.org>.
lujiajing1126 commented on pull request #78:
URL: https://github.com/apache/skywalking-banyandb/pull/78#issuecomment-1038150088
I confirm that this branch significantly reduce the memory usage. But some issues,
<img width="1330" alt="image" src="https://user-images.githubusercontent.com/2568208/153757321-9c0d16ef-8d1a-4aaf-8fcf-7b7ca4ffe5cc.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-banyandb] hanahmily commented on pull request #78: Extend tsdb to contain streams of a group
Posted by GitBox <gi...@apache.org>.
hanahmily commented on pull request #78:
URL: https://github.com/apache/skywalking-banyandb/pull/78#issuecomment-1038783132
@lujiajing1126 I tried to fix this issue, pls verify my patch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-banyandb] lujiajing1126 merged pull request #78: Extend tsdb to contain streams of a group
Posted by GitBox <gi...@apache.org>.
lujiajing1126 merged pull request #78:
URL: https://github.com/apache/skywalking-banyandb/pull/78
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org