You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "xiaokang (via GitHub)" <gi...@apache.org> on 2023/02/09 10:05:09 UTC
[GitHub] [doris] xiaokang opened a new pull request, #16578: [Improvement](inverted index) inverted index query match bitmap cache
xiaokang opened a new pull request, #16578:
URL: https://github.com/apache/doris/pull/16578
# Proposed changes
Issue Number: close #xxx
## Problem summary
Add cache for inverted index query match bitmap to accelerate common query keyword, especially for keyword matching many rows.
Tests result:
- large result: matching 99% out of 247 million rows shows 8x speed up.
- small result: matching 0.1% out of 247 million rows shows 2x speed up.
## Checklist(Required)
1. Does it affect the original behavior:
- [ ] Yes
- [ ] No
- [ ] I don't know
2. Has unit tests been added:
- [ ] Yes
- [ ] No
- [ ] No Need
3. Has document been added or modified:
- [ ] Yes
- [ ] No
- [ ] No Need
4. Does it need to update dependencies:
- [ ] Yes
- [ ] No
5. Are there any changes that cannot be rolled back:
- [ ] Yes (If Yes, please explain WHY)
- [ ] No
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] Tanya-W commented on pull request #16578: [Improvement](inverted index) inverted index query match bitmap cache
Posted by "Tanya-W (via GitHub)" <gi...@apache.org>.
Tanya-W commented on PR #16578:
URL: https://github.com/apache/doris/pull/16578#issuecomment-1425797407
LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morningman merged pull request #16578: [Improvement](inverted index) inverted index query match bitmap cache
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman merged PR #16578:
URL: https://github.com/apache/doris/pull/16578
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16578: [Improvement](inverted index) inverted index query match bitmap cache
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #16578:
URL: https://github.com/apache/doris/pull/16578#issuecomment-1425800543
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on a diff in pull request #16578: [Improvement](inverted index) inverted index query match bitmap cache
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #16578:
URL: https://github.com/apache/doris/pull/16578#discussion_r1101238781
##########
be/src/olap/rowset/segment_v2/inverted_index_cache.h:
##########
@@ -163,5 +165,105 @@ class InvertedIndexCacheHandle {
DISALLOW_COPY_AND_ASSIGN(InvertedIndexCacheHandle);
};
+
+
+enum class InvertedIndexQueryType;
+
+class InvertedIndexQueryCacheHandle;
+
+class InvertedIndexQueryCache {
+public:
+ // cache key
+ struct CacheKey {
+ io::Path index_path; // index file path
+ std::string column_name; // column name
+ InvertedIndexQueryType query_type; // query type
+ std::wstring value; // query value
+
+ // Encode to a flat binary which can be used as LRUCache's key
+ std::string encode() const {
+ std::string key_buf(index_path.string());
+ key_buf.append("/");
+ key_buf.append(column_name);
+ key_buf.append("/");
+ key_buf.append(1, static_cast<char>(query_type));
+ key_buf.append("/");
+ key_buf.append(lucene::util::Misc::toString(value.c_str()));
+ return key_buf;
+ }
+ };
+
+ using CacheValue = roaring::Roaring;
+
+ // Create global instance of this class
+ static void create_global_cache(size_t capacity, int32_t index_cache_percentage,
+ uint32_t num_shards = 16) {
+ DCHECK(_s_instance == nullptr);
+ static InvertedIndexQueryCache instance(capacity, index_cache_percentage, num_shards);
+ _s_instance = &instance;
+ }
+
+ // Return global instance.
+ // Client should call create_global_cache before.
+ static InvertedIndexQueryCache* instance() { return _s_instance; }
+
+ InvertedIndexQueryCache() = delete;
+
+ InvertedIndexQueryCache(size_t capacity, int32_t index_cache_percentage, uint32_t num_shards) {
+ _cache = std::unique_ptr<Cache>(
+ new_lru_cache("InvertedIndexQueryCache", capacity, LRUCacheType::SIZE, num_shards));
+ }
+
+ bool lookup(const CacheKey& key, InvertedIndexQueryCacheHandle* handle);
+
+ void insert(const CacheKey& key, roaring::Roaring* bitmap,
+ InvertedIndexQueryCacheHandle* handle);
+
+private:
+ static InvertedIndexQueryCache* _s_instance;
+ std::unique_ptr<Cache> _cache {nullptr};
+};
+
+class InvertedIndexQueryCacheHandle {
+public:
+ InvertedIndexQueryCacheHandle() {}
Review Comment:
warning: use '= default' to define a trivial default constructor [modernize-use-equals-default]
```suggestion
InvertedIndexQueryCacheHandle() = default;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on a diff in pull request #16578: [Improvement](inverted index) inverted index query match bitmap cache
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #16578:
URL: https://github.com/apache/doris/pull/16578#discussion_r1102188127
##########
be/test/olap/rowset/segment_v2/inverted_index_searcher_cache_test.cpp:
##########
@@ -115,14 +117,14 @@ TEST_F(InvertedIndexSearcherCacheTest, insert_lookup) {
// not use cache
InvertedIndexCacheHandle inverted_index_cache_handle_2;
status = index_searcher_cache->get_index_searcher(fs, kTestDir, file_name_not_exist_2,
- &inverted_index_cache_handle_2, false);
+ &inverted_index_cache_handle_2, &stats, false);
EXPECT_EQ(Status::OK(), status);
EXPECT_TRUE(inverted_index_cache_handle_2.owned);
EXPECT_EQ(nullptr, inverted_index_cache_handle_2._cache);
Review Comment:
warning: '_cache' is a private member of 'doris::segment_v2::InvertedIndexCacheHandle' [clang-diagnostic-error]
```cpp
EXPECT_EQ(nullptr, inverted_index_cache_handle_2._cache);
^
```
**be/src/olap/rowset/segment_v2/inverted_index_cache.h:159:** declared private here
```cpp
Cache* _cache = nullptr;
^
```
##########
be/test/olap/rowset/segment_v2/inverted_index_searcher_cache_test.cpp:
##########
@@ -115,14 +117,14 @@
// not use cache
InvertedIndexCacheHandle inverted_index_cache_handle_2;
status = index_searcher_cache->get_index_searcher(fs, kTestDir, file_name_not_exist_2,
- &inverted_index_cache_handle_2, false);
+ &inverted_index_cache_handle_2, &stats, false);
EXPECT_EQ(Status::OK(), status);
EXPECT_TRUE(inverted_index_cache_handle_2.owned);
EXPECT_EQ(nullptr, inverted_index_cache_handle_2._cache);
EXPECT_EQ(nullptr, inverted_index_cache_handle_2._handle);
Review Comment:
warning: '_handle' is a private member of 'doris::segment_v2::InvertedIndexCacheHandle' [clang-diagnostic-error]
```cpp
EXPECT_EQ(nullptr, inverted_index_cache_handle_2._handle);
^
```
**be/src/olap/rowset/segment_v2/inverted_index_cache.h:160:** declared private here
```cpp
Cache::Handle* _handle = nullptr;
^
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org