You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tubemq.apache.org by GitBox <gi...@apache.org> on 2020/06/09 16:14:19 UTC
[GitHub] [incubator-tubemq] hystericalhell opened a new pull request #143: [TUBEMQ-124] Structured index storage
hystericalhell opened a new pull request #143:
URL: https://github.com/apache/incubator-tubemq/pull/143
**Initial Prototype** for demonstration only
Further details needs to be discussed. Currently it should be incomplete, and it *will* be some time.
Here the major issue directly from initial plan in TUBEMQ-124, I think, contains (but not limited to):
1. Bloom Filter, as a probability prediction set, does **not** support a **remove** operation, so we have to choose an *appropriate* time to rebuild it after a long run with massive topics in and out, which is such an expensive thing just like the GC STP (stop-the-world);
2. Fine-grained control might be introduced to scatter the impact while exploit the negative-prediction efficiency, like we could do something like Java 8 revised ConcurrentHashMap, we could setup Bloom Filter on Segment (even available for memory prediction), and apply something like URA (unbalanced resource allocation) to better concentrate messages of the same topic into the same segment.
Please review the prototype about the overall architecture of how Bloom Filter (my opinion) could behave, and let's discuss something in details about above factors, if any or more.
Details: [TUBEMQ-124](https://issues.apache.org/jira/projects/TUBEMQ/issues/TUBEMQ-124)
*The same would be synchronized to original issue comments*
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tubemq] codecov-commenter commented on pull request #143: [TUBEMQ-124] Structured index storage
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on pull request #143:
URL: https://github.com/apache/incubator-tubemq/pull/143#issuecomment-643959751
# [Codecov](https://codecov.io/gh/apache/incubator-tubemq/pull/143?src=pr&el=h1) Report
> :exclamation: No coverage uploaded for pull request base (`master@b2ddb28`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit).
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/incubator-tubemq/pull/143?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #143 +/- ##
========================================
Coverage ? 7.59%
Complexity ? 436
========================================
Files ? 235
Lines ? 26621
Branches ? 4443
========================================
Hits ? 2023
Misses ? 24151
Partials ? 447
```
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-tubemq/pull/143?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-tubemq/pull/143?src=pr&el=footer). Last update [b2ddb28...cd32f2f](https://codecov.io/gh/apache/incubator-tubemq/pull/143?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tubemq] hystericalhell commented on pull request #143: [TUBEMQ-124] Structured index storage
Posted by GitBox <gi...@apache.org>.
hystericalhell commented on pull request #143:
URL: https://github.com/apache/incubator-tubemq/pull/143#issuecomment-645425892
Add IndexScale control in config file, while resolve estimated scale issue by accepting parameter from user.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tubemq] gosonzhang commented on pull request #143: [TUBEMQ-124] Structured index storage
Posted by GitBox <gi...@apache.org>.
gosonzhang commented on pull request #143:
URL: https://github.com/apache/incubator-tubemq/pull/143#issuecomment-762160493
I'll close this pr, thanks @hystericalhell
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tubemq] hystericalhell edited a comment on pull request #143: [TUBEMQ-124] Structured index storage
Posted by GitBox <gi...@apache.org>.
hystericalhell edited a comment on pull request #143:
URL: https://github.com/apache/incubator-tubemq/pull/143#issuecomment-643934089
Updated:
As discussed with Guocheng Zhang, there are a few facts current system has implemented:
1. A SegmentList consists of only ONE topic, the filter is actually to distinguish clients;
2. Messages in Segments are naturally sequenced by time, which means there's only APPEND operation occurs for messages;
3. Segment would always be detached and removed when it's Expired, and it would be removed as a whole, not any part of it or a few messages in it.
Currently it's still on-going, current submission is still an intermediate solution.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tubemq] hystericalhell commented on pull request #143: [TUBEMQ-124] Structured index storage
Posted by GitBox <gi...@apache.org>.
hystericalhell commented on pull request #143:
URL: https://github.com/apache/incubator-tubemq/pull/143#issuecomment-643934089
Updated:
As discussed with Guocheng Zhang, there are a few facts current system has implemented:
1. A SegmentList consists of only ONE topic, the filter is actually to distinguish clients;
2. Messages in Segments are naturally sequenced by time, which means there's only APPEND operation occurs for messages;
3. Segment would always be detached and removed when it's Expired, and it would be removed as a whole, not any part of it or a few messages in it.
Currently it's still on-going, current submission is still an intermediate solution.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tubemq] gosonzhang closed pull request #143: [TUBEMQ-124] Structured index storage
Posted by GitBox <gi...@apache.org>.
gosonzhang closed pull request #143:
URL: https://github.com/apache/incubator-tubemq/pull/143
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org