You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "Tanya-W (via GitHub)" <gi...@apache.org> on 2023/03/22 11:26:49 UTC
[GitHub] [doris] Tanya-W opened a new pull request, #18033: [Refactor](inverted index cache) Use asc set instead of priority queue at the lru cache
Tanya-W opened a new pull request, #18033:
URL: https://github.com/apache/doris/pull/18033
# Proposed changes
Issue Number: close #xxx
## Problem summary
use asc set instead of priority queue at the LRU cache, to keep the lifecycle of the LRUHandle consistent in the sorted set and the LUR free list
## Checklist(Required)
* [ ] Does it affect the original behavior
* [ ] Has unit tests been added
* [ ] Has document been added or modified
* [ ] Does it need to update dependencies
* [ ] Is this PR support rollback (If NO, please explain WHY)
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] xiaokang commented on a diff in pull request #18033: [Refactor](inverted index cache) Use asc set instead of priority queue at the lru cache
Posted by "xiaokang (via GitHub)" <gi...@apache.org>.
xiaokang commented on code in PR #18033:
URL: https://github.com/apache/doris/pull/18033#discussion_r1144825856
##########
be/src/olap/lru_cache.h:
##########
@@ -314,13 +314,10 @@ class HandleTable {
void _resize();
};
-// pair first is timestatmp, put <timestatmp, LRUHandle*> into asc priority_queue,
-// when need to free space, can first evict the top of the LRUHandleHeap,
-// because the top element's timestamp is the oldest.
-typedef std::priority_queue<std::pair<int64_t, LRUHandle*>,
- std::vector<std::pair<int64_t, LRUHandle*>>,
- std::greater<std::pair<int64_t, LRUHandle*>>>
- LRUHandleHeap;
+// pair first is timestatmp, put <timestatmp, LRUHandle*> into asc set,
+// when need to free space, can first evict the begin of the set,
+// because the begin element's timestamp is the oldest.
+typedef std::set<std::pair<int64_t, LRUHandle*>> LRUHandleAscSet;
Review Comment:
using LRUHandleSortedSet = std::set<std::pair<int64_t, LRUHandle*>>;
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #18033: [Refactor](inverted index cache) Use asc set instead of priority queue at the lru cache
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #18033:
URL: https://github.com/apache/doris/pull/18033#issuecomment-1480537460
TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 34.7 seconds
stream load tsv: 447 seconds loaded 74807831229 Bytes, about 159 MB/s
stream load json: 24 seconds loaded 2358488459 Bytes, about 93 MB/s
stream load orc: 73 seconds loaded 1101869774 Bytes, about 14 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230323030318_clickbench_pr_119109.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] Tanya-W commented on pull request #18033: [Refactor](inverted index cache) Use asc set instead of priority queue at the lru cache
Posted by "Tanya-W (via GitHub)" <gi...@apache.org>.
Tanya-W commented on PR #18033:
URL: https://github.com/apache/doris/pull/18033#issuecomment-1479628011
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] Tanya-W commented on pull request #18033: [Refactor](inverted index cache) Use asc set instead of priority queue at the lru cache
Posted by "Tanya-W (via GitHub)" <gi...@apache.org>.
Tanya-W commented on PR #18033:
URL: https://github.com/apache/doris/pull/18033#issuecomment-1481402289
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #18033: [Refactor](inverted index cache) Use asc set instead of priority queue at the lru cache
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #18033:
URL: https://github.com/apache/doris/pull/18033#issuecomment-1482327070
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #18033: [Refactor](inverted index cache) Use asc set instead of priority queue at the lru cache
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #18033:
URL: https://github.com/apache/doris/pull/18033#issuecomment-1481407754
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #18033: [Refactor](inverted index cache) Use asc set instead of priority queue at the lru cache
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #18033:
URL: https://github.com/apache/doris/pull/18033#issuecomment-1479595824
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #18033: [Refactor](inverted index cache) Use asc set instead of priority queue at the lru cache
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #18033:
URL: https://github.com/apache/doris/pull/18033#issuecomment-1480541290
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #18033: [Refactor](inverted index cache) Use asc set instead of priority queue at the lru cache
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #18033:
URL: https://github.com/apache/doris/pull/18033#issuecomment-1479633361
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] Tanya-W commented on pull request #18033: [Refactor](inverted index cache) Use asc set instead of priority queue at the lru cache
Posted by "Tanya-W (via GitHub)" <gi...@apache.org>.
Tanya-W commented on PR #18033:
URL: https://github.com/apache/doris/pull/18033#issuecomment-1480539330
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] Tanya-W commented on pull request #18033: [Refactor](inverted index cache) Use asc set instead of priority queue at the lru cache
Posted by "Tanya-W (via GitHub)" <gi...@apache.org>.
Tanya-W commented on PR #18033:
URL: https://github.com/apache/doris/pull/18033#issuecomment-1480896664
run p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] qidaye merged pull request #18033: [Refactor](inverted index cache) Use asc set instead of priority queue at the lru cache
Posted by "qidaye (via GitHub)" <gi...@apache.org>.
qidaye merged PR #18033:
URL: https://github.com/apache/doris/pull/18033
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #18033: [Refactor](inverted index cache) Use asc set instead of priority queue at the lru cache
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #18033:
URL: https://github.com/apache/doris/pull/18033#issuecomment-1482410687
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] Tanya-W commented on pull request #18033: [Refactor](inverted index cache) Use asc set instead of priority queue at the lru cache
Posted by "Tanya-W (via GitHub)" <gi...@apache.org>.
Tanya-W commented on PR #18033:
URL: https://github.com/apache/doris/pull/18033#issuecomment-1482407479
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on a diff in pull request #18033: [Refactor](inverted index cache) Use asc set instead of priority queue at the lru cache
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #18033:
URL: https://github.com/apache/doris/pull/18033#discussion_r1144657067
##########
be/src/olap/lru_cache.h:
##########
@@ -314,13 +314,10 @@ class HandleTable {
void _resize();
};
-// pair first is timestatmp, put <timestatmp, LRUHandle*> into asc priority_queue,
-// when need to free space, can first evict the top of the LRUHandleHeap,
-// because the top element's timestamp is the oldest.
-typedef std::priority_queue<std::pair<int64_t, LRUHandle*>,
- std::vector<std::pair<int64_t, LRUHandle*>>,
- std::greater<std::pair<int64_t, LRUHandle*>>>
- LRUHandleHeap;
+// pair first is timestatmp, put <timestatmp, LRUHandle*> into asc set,
+// when need to free space, can first evict the begin of the set,
+// because the begin element's timestamp is the oldest.
+typedef std::set<std::pair<int64_t, LRUHandle*>> LRUHandleAscSet;
Review Comment:
warning: use 'using' instead of 'typedef' [modernize-use-using]
```suggestion
using LRUHandleAscSet = std::set<std::pair<int64_t, LRUHandle *>>;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org