You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "eldenmoon (via GitHub)" <gi...@apache.org> on 2024/02/28 11:54:16 UTC
[PR] [cloud](topn-opt) support topn 2 phase read in cloud mode [doris]
eldenmoon opened a new pull request, #31548:
URL: https://github.com/apache/doris/pull/31548
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [cloud](topn-opt) support topn 2 phase read in cloud mode [doris]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on PR #31548:
URL: https://github.com/apache/doris/pull/31548#issuecomment-1970329507
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [cloud](topn-opt) support topn 2 phase read in cloud mode [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #31548:
URL: https://github.com/apache/doris/pull/31548#discussion_r1506945846
##########
be/src/exec/rowid_fetcher.cpp:
##########
@@ -275,4 +284,192 @@ Status RowIDFetcher::fetch(const vectorized::ColumnPtr& column_row_ids,
return Status::OK();
}
+template <typename Func>
+auto scope_timer_run(Func fn, int64_t* cost) -> decltype(fn()) {
+ MonotonicStopWatch watch;
+ watch.start();
+ auto res = fn();
+ *cost += watch.elapsed_time() / 1000 / 1000;
+ return res;
+}
+
+struct IteratorKey {
+ int64_t tablet_id;
+ RowsetId rowset_id;
+ uint64_t segment_id;
+ int slot_id;
+
+ // unordered map std::equal_to
+ bool operator==(const IteratorKey& rhs) const {
+ return tablet_id == rhs.tablet_id && rowset_id == rhs.rowset_id &&
+ segment_id == rhs.segment_id && slot_id == rhs.slot_id;
+ }
+};
+
+struct HashOfIteratorKey {
+ size_t operator()(const IteratorKey& key) const {
+ size_t seed = 0;
+ seed = HashUtil::hash64(&key.tablet_id, sizeof(key.tablet_id), seed);
+ seed = HashUtil::hash64(&key.rowset_id.hi, sizeof(key.rowset_id.hi), seed);
+ seed = HashUtil::hash64(&key.rowset_id.mi, sizeof(key.rowset_id.mi), seed);
+ seed = HashUtil::hash64(&key.rowset_id.lo, sizeof(key.rowset_id.lo), seed);
+ seed = HashUtil::hash64(&key.segment_id, sizeof(key.segment_id), seed);
+ seed = HashUtil::hash64(&key.slot_id, sizeof(key.slot_id), seed);
+ return seed;
+ }
+};
+
+struct IteratorItem {
+ std::unique_ptr<ColumnIterator> iterator;
+ // for holding the reference of segment to avoid use after release
+ SegmentSharedPtr segment;
+};
+
+Status RowIdStorageReader::read_by_rowids(const PMultiGetRequest& request,
Review Comment:
warning: function 'read_by_rowids' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
Status RowIdStorageReader::read_by_rowids(const PMultiGetRequest& request,
^
```
<details>
<summary>Additional context</summary>
**be/src/exec/rowid_fetcher.cpp:327:** 144 lines including whitespace and comments (threshold 80)
```cpp
Status RowIdStorageReader::read_by_rowids(const PMultiGetRequest& request,
^
```
</details>
##########
be/src/service/internal_service.cpp:
##########
@@ -896,11 +878,11 @@ struct AsyncRPCContext {
brpc::CallId cid;
};
-void PInternalServiceImpl::fetch_remote_tablet_schema(google::protobuf::RpcController* controller,
- const PFetchRemoteSchemaRequest* request,
- PFetchRemoteSchemaResponse* response,
- google::protobuf::Closure* done) {
- bool ret = _heavy_work_pool.try_offer([this, request, response, done]() {
+void PInternalService::fetch_remote_tablet_schema(google::protobuf::RpcController* controller,
Review Comment:
warning: method 'fetch_remote_tablet_schema' can be made static [readability-convert-member-functions-to-static]
```suggestion
static void PInternalService::fetch_remote_tablet_schema(google::protobuf::RpcController* controller,
```
##########
be/src/service/internal_service.cpp:
##########
@@ -896,11 +878,11 @@
brpc::CallId cid;
};
-void PInternalServiceImpl::fetch_remote_tablet_schema(google::protobuf::RpcController* controller,
- const PFetchRemoteSchemaRequest* request,
- PFetchRemoteSchemaResponse* response,
- google::protobuf::Closure* done) {
- bool ret = _heavy_work_pool.try_offer([this, request, response, done]() {
+void PInternalService::fetch_remote_tablet_schema(google::protobuf::RpcController* controller,
Review Comment:
warning: function 'fetch_remote_tablet_schema' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
void PInternalService::fetch_remote_tablet_schema(google::protobuf::RpcController* controller,
^
```
<details>
<summary>Additional context</summary>
**be/src/service/internal_service.cpp:880:** 93 lines including whitespace and comments (threshold 80)
```cpp
void PInternalService::fetch_remote_tablet_schema(google::protobuf::RpcController* controller,
^
```
</details>
##########
be/src/service/internal_service.cpp:
##########
@@ -1738,201 +1718,17 @@
}
}
-template <typename Func>
-auto scope_timer_run(Func fn, int64_t* cost) -> decltype(fn()) {
- MonotonicStopWatch watch;
- watch.start();
- auto res = fn();
- *cost += watch.elapsed_time() / 1000 / 1000;
- return res;
-}
-
-struct IteratorKey {
- int64_t tablet_id;
- RowsetId rowset_id;
- uint64_t segment_id;
- int slot_id;
-
- // unordered map std::equal_to
- bool operator==(const IteratorKey& rhs) const {
- return tablet_id == rhs.tablet_id && rowset_id == rhs.rowset_id &&
- segment_id == rhs.segment_id && slot_id == rhs.slot_id;
- }
-};
-
-struct HashOfIteratorKey {
- size_t operator()(const IteratorKey& key) const {
- size_t seed = 0;
- seed = HashUtil::hash64(&key.tablet_id, sizeof(key.tablet_id), seed);
- seed = HashUtil::hash64(&key.rowset_id.hi, sizeof(key.rowset_id.hi), seed);
- seed = HashUtil::hash64(&key.rowset_id.mi, sizeof(key.rowset_id.mi), seed);
- seed = HashUtil::hash64(&key.rowset_id.lo, sizeof(key.rowset_id.lo), seed);
- seed = HashUtil::hash64(&key.segment_id, sizeof(key.segment_id), seed);
- seed = HashUtil::hash64(&key.slot_id, sizeof(key.slot_id), seed);
- return seed;
- }
-};
-
-struct IteratorItem {
- std::unique_ptr<ColumnIterator> iterator;
- // for holding the reference of segment to avoid use after release
- SegmentSharedPtr segment;
-};
-
-Status PInternalServiceImpl::_multi_get(const PMultiGetRequest& request,
- PMultiGetResponse* response) {
- OlapReaderStatistics stats;
- vectorized::Block result_block;
- int64_t acquire_tablet_ms = 0;
- int64_t acquire_rowsets_ms = 0;
- int64_t acquire_segments_ms = 0;
- int64_t lookup_row_data_ms = 0;
-
- // init desc
- TupleDescriptor desc(request.desc());
- std::vector<SlotDescriptor> slots;
- slots.reserve(request.slots().size());
- for (const auto& pslot : request.slots()) {
- slots.push_back(SlotDescriptor(pslot));
- desc.add_slot(&slots.back());
- }
-
- // init read schema
- TabletSchema full_read_schema;
- for (const ColumnPB& column_pb : request.column_desc()) {
- full_read_schema.append_column(TabletColumn(column_pb));
- }
-
- std::unordered_map<IteratorKey, IteratorItem, HashOfIteratorKey> iterator_map;
- // read row by row
- for (size_t i = 0; i < request.row_locs_size(); ++i) {
- const auto& row_loc = request.row_locs(i);
- MonotonicStopWatch watch;
- watch.start();
- TabletSharedPtr tablet = scope_timer_run(
- [&]() {
- return _engine.tablet_manager()->get_tablet(row_loc.tablet_id(),
- true /*include deleted*/);
- },
- &acquire_tablet_ms);
- RowsetId rowset_id;
- rowset_id.init(row_loc.rowset_id());
- if (!tablet) {
- continue;
- }
- // We ensured it's rowset is not released when init Tablet reader param, rowset->update_delayed_expired_timestamp();
- BetaRowsetSharedPtr rowset = std::static_pointer_cast<BetaRowset>(scope_timer_run(
- [&]() { return _engine.get_quering_rowset(rowset_id); }, &acquire_rowsets_ms));
- if (!rowset) {
- LOG(INFO) << "no such rowset " << rowset_id;
- continue;
- }
- size_t row_size = 0;
- Defer _defer([&]() {
- LOG_EVERY_N(INFO, 100)
- << "multiget_data single_row, cost(us):" << watch.elapsed_time() / 1000
- << ", row_size:" << row_size;
- *response->add_row_locs() = row_loc;
- });
- SegmentCacheHandle segment_cache;
- RETURN_IF_ERROR(scope_timer_run(
- [&]() {
- return SegmentLoader::instance()->load_segments(rowset, &segment_cache, true);
- },
- &acquire_segments_ms));
- // find segment
- auto it = std::find_if(segment_cache.get_segments().cbegin(),
- segment_cache.get_segments().cend(),
- [&row_loc](const segment_v2::SegmentSharedPtr& seg) {
- return seg->id() == row_loc.segment_id();
- });
- if (it == segment_cache.get_segments().end()) {
- continue;
- }
- segment_v2::SegmentSharedPtr segment = *it;
- GlobalRowLoacation row_location(row_loc.tablet_id(), rowset->rowset_id(),
- row_loc.segment_id(), row_loc.ordinal_id());
- // fetch by row store, more effcient way
- if (request.fetch_row_store()) {
- CHECK(tablet->tablet_schema()->store_row_column());
- RowLocation loc(rowset_id, segment->id(), row_loc.ordinal_id());
- string* value = response->add_binary_row_data();
- RETURN_IF_ERROR(scope_timer_run(
- [&]() {
- return tablet->lookup_row_data({}, loc, rowset, &desc, stats, *value);
- },
- &lookup_row_data_ms));
- row_size = value->size();
- continue;
- }
-
- // fetch by column store
- if (result_block.is_empty_column()) {
- result_block = vectorized::Block(desc.slots(), request.row_locs().size());
- }
- VLOG_DEBUG << "Read row location "
- << fmt::format("{}, {}, {}, {}", row_location.tablet_id,
- row_location.row_location.rowset_id.to_string(),
- row_location.row_location.segment_id,
- row_location.row_location.row_id);
- for (int x = 0; x < desc.slots().size(); ++x) {
- auto row_id = static_cast<segment_v2::rowid_t>(row_loc.ordinal_id());
- vectorized::MutableColumnPtr column =
- result_block.get_by_position(x).column->assume_mutable();
- IteratorKey iterator_key {.tablet_id = tablet->tablet_id(),
- .rowset_id = rowset_id,
- .segment_id = row_loc.segment_id(),
- .slot_id = desc.slots()[x]->id()};
- IteratorItem& iterator_item = iterator_map[iterator_key];
- if (iterator_item.segment == nullptr) {
- // hold the reference
- iterator_map[iterator_key].segment = segment;
- }
- segment = iterator_item.segment;
- RETURN_IF_ERROR(segment->seek_and_read_by_rowid(full_read_schema, desc.slots()[x],
- row_id, column, stats,
- iterator_item.iterator));
- }
- }
- // serialize block if not empty
- if (!result_block.is_empty_column()) {
- VLOG_DEBUG << "dump block:" << result_block.dump_data(0, 10)
- << ", be_exec_version:" << request.be_exec_version();
- [[maybe_unused]] size_t compressed_size = 0;
- [[maybe_unused]] size_t uncompressed_size = 0;
- int be_exec_version = request.has_be_exec_version() ? request.be_exec_version() : 0;
- RETURN_IF_ERROR(result_block.serialize(be_exec_version, response->mutable_block(),
- &uncompressed_size, &compressed_size,
- segment_v2::CompressionTypePB::LZ4));
- }
-
- LOG(INFO) << "Query stats: "
- << fmt::format(
- "hit_cached_pages:{}, total_pages_read:{}, compressed_bytes_read:{}, "
- "io_latency:{}ns, "
- "uncompressed_bytes_read:{},"
- "bytes_read:{},"
- "acquire_tablet_ms:{}, acquire_rowsets_ms:{}, acquire_segments_ms:{}, "
- "lookup_row_data_ms:{}",
- stats.cached_pages_num, stats.total_pages_num, stats.compressed_bytes_read,
- stats.io_ns, stats.uncompressed_bytes_read, stats.bytes_read,
- acquire_tablet_ms, acquire_rowsets_ms, acquire_segments_ms,
- lookup_row_data_ms);
- return Status::OK();
-}
-
-void PInternalServiceImpl::multiget_data(google::protobuf::RpcController* controller,
- const PMultiGetRequest* request,
- PMultiGetResponse* response,
- google::protobuf::Closure* done) {
- bool ret = _light_work_pool.try_offer([request, response, done, this]() {
+void PInternalService::multiget_data(google::protobuf::RpcController* controller,
Review Comment:
warning: method 'multiget_data' can be made static [readability-convert-member-functions-to-static]
```suggestion
static void PInternalService::multiget_data(google::protobuf::RpcController* controller,
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [cloud](topn-opt) support topn 2 phase read in cloud mode [doris]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on code in PR #31548:
URL: https://github.com/apache/doris/pull/31548#discussion_r1506911569
##########
be/src/olap/olap_server.cpp:
##########
@@ -129,6 +129,12 @@ Status StorageEngine::start_bg_threads() {
&_unused_rowset_monitor_thread));
LOG(INFO) << "unused rowset monitor thread started";
+ RETURN_IF_ERROR(Thread::create(
Review Comment:
not special, it's for both local and cloud mode to clean expired quering rowsets
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [cloud](topn-opt) support topn 2 phase read in cloud mode [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31548:
URL: https://github.com/apache/doris/pull/31548#issuecomment-1968971539
TeamCity be ut coverage result:
Function Coverage: 35.68% (8547/23955)
Line Coverage: 27.47% (69362/252467)
Region Coverage: 26.63% (35957/135024)
Branch Coverage: 23.44% (18382/78408)
Coverage Report: http://coverage.selectdb-in.cc/coverage/5f662e4ccaef51e09880109e45b5b1dab6a9ad5e_5f662e4ccaef51e09880109e45b5b1dab6a9ad5e/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [cloud](topn-opt) support topn 2 phase read in cloud mode [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #31548:
URL: https://github.com/apache/doris/pull/31548#discussion_r1505843497
##########
be/src/exec/rowid_fetcher.cpp:
##########
@@ -275,4 +284,197 @@ Status RowIDFetcher::fetch(const vectorized::ColumnPtr& column_row_ids,
return Status::OK();
}
+template <typename Func>
+auto scope_timer_run(Func fn, int64_t* cost) -> decltype(fn()) {
+ MonotonicStopWatch watch;
+ watch.start();
+ auto res = fn();
+ *cost += watch.elapsed_time() / 1000 / 1000;
+ return res;
+}
+
+struct IteratorKey {
+ int64_t tablet_id;
+ RowsetId rowset_id;
+ uint64_t segment_id;
+ int slot_id;
+
+ // unordered map std::equal_to
+ bool operator==(const IteratorKey& rhs) const {
+ return tablet_id == rhs.tablet_id && rowset_id == rhs.rowset_id &&
+ segment_id == rhs.segment_id && slot_id == rhs.slot_id;
+ }
+};
+
+struct HashOfIteratorKey {
+ size_t operator()(const IteratorKey& key) const {
+ size_t seed = 0;
+ seed = HashUtil::hash64(&key.tablet_id, sizeof(key.tablet_id), seed);
+ seed = HashUtil::hash64(&key.rowset_id.hi, sizeof(key.rowset_id.hi), seed);
+ seed = HashUtil::hash64(&key.rowset_id.mi, sizeof(key.rowset_id.mi), seed);
+ seed = HashUtil::hash64(&key.rowset_id.lo, sizeof(key.rowset_id.lo), seed);
+ seed = HashUtil::hash64(&key.segment_id, sizeof(key.segment_id), seed);
+ seed = HashUtil::hash64(&key.slot_id, sizeof(key.slot_id), seed);
+ return seed;
+ }
+};
+
+struct IteratorItem {
+ std::unique_ptr<ColumnIterator> iterator;
+ // for holding the reference of segment to avoid use after release
+ SegmentSharedPtr segment;
+};
+
+Status RowIdStorageReader::read_by_rowids(const PMultiGetRequest& request,
Review Comment:
warning: method 'read_by_rowids' can be made static [readability-convert-member-functions-to-static]
```suggestion
static Status RowIdStorageReader::read_by_rowids(const PMultiGetRequest& request,
```
##########
be/src/olap/storage_engine.cpp:
##########
@@ -1396,12 +1404,12 @@ Status StorageEngine::get_compaction_status_json(std::string* result) {
return Status::OK();
}
-void StorageEngine::add_quering_rowset(RowsetSharedPtr rs) {
+void BaseStorageEngine::add_quering_rowset(RowsetSharedPtr rs) {
std::lock_guard<std::mutex> lock(_quering_rowsets_mutex);
_querying_rowsets.emplace(rs->rowset_id(), rs);
}
-RowsetSharedPtr StorageEngine::get_quering_rowset(RowsetId rs_id) {
+RowsetSharedPtr BaseStorageEngine::get_quering_rowset(RowsetId rs_id) {
Review Comment:
warning: method 'get_quering_rowset' can be made static [readability-convert-member-functions-to-static]
be/src/olap/storage_engine.h:133:
```diff
- RowsetSharedPtr get_quering_rowset(RowsetId rs_id);
+ static RowsetSharedPtr get_quering_rowset(RowsetId rs_id);
```
##########
be/src/olap/storage_engine.cpp:
##########
@@ -1410,9 +1418,19 @@
return nullptr;
}
-void StorageEngine::evict_querying_rowset(RowsetId rs_id) {
- std::lock_guard<std::mutex> lock(_quering_rowsets_mutex);
- _querying_rowsets.erase(rs_id);
+void BaseStorageEngine::_evict_querying_rowset() {
Review Comment:
warning: method '_evict_querying_rowset' can be made static [readability-convert-member-functions-to-static]
be/src/olap/storage_engine.h:136:
```diff
- void _evict_querying_rowset();
+ static void _evict_querying_rowset();
```
##########
be/src/cloud/cloud_internal_service.cpp:
##########
@@ -17,11 +17,39 @@
#include "cloud/cloud_internal_service.h"
+#include <brpc/closure_guard.h>
+
+#include "cloud/cloud_storage_engine.h"
+#include "common/signal_handler.h"
+#include "exec/rowid_fetcher.h"
+
namespace doris {
CloudInternalServiceImpl::CloudInternalServiceImpl(CloudStorageEngine& engine, ExecEnv* exec_env)
: PInternalService(exec_env), _engine(engine) {}
CloudInternalServiceImpl::~CloudInternalServiceImpl() = default;
+void CloudInternalServiceImpl::multiget_data(google::protobuf::RpcController* controller,
Review Comment:
warning: method 'multiget_data' can be made static [readability-convert-member-functions-to-static]
```suggestion
static void CloudInternalServiceImpl::multiget_data(google::protobuf::RpcController* controller,
```
##########
be/src/exec/rowid_fetcher.cpp:
##########
@@ -275,4 +284,197 @@
return Status::OK();
}
+template <typename Func>
+auto scope_timer_run(Func fn, int64_t* cost) -> decltype(fn()) {
+ MonotonicStopWatch watch;
+ watch.start();
+ auto res = fn();
+ *cost += watch.elapsed_time() / 1000 / 1000;
+ return res;
+}
+
+struct IteratorKey {
+ int64_t tablet_id;
+ RowsetId rowset_id;
+ uint64_t segment_id;
+ int slot_id;
+
+ // unordered map std::equal_to
+ bool operator==(const IteratorKey& rhs) const {
+ return tablet_id == rhs.tablet_id && rowset_id == rhs.rowset_id &&
+ segment_id == rhs.segment_id && slot_id == rhs.slot_id;
+ }
+};
+
+struct HashOfIteratorKey {
+ size_t operator()(const IteratorKey& key) const {
+ size_t seed = 0;
+ seed = HashUtil::hash64(&key.tablet_id, sizeof(key.tablet_id), seed);
+ seed = HashUtil::hash64(&key.rowset_id.hi, sizeof(key.rowset_id.hi), seed);
+ seed = HashUtil::hash64(&key.rowset_id.mi, sizeof(key.rowset_id.mi), seed);
+ seed = HashUtil::hash64(&key.rowset_id.lo, sizeof(key.rowset_id.lo), seed);
+ seed = HashUtil::hash64(&key.segment_id, sizeof(key.segment_id), seed);
+ seed = HashUtil::hash64(&key.slot_id, sizeof(key.slot_id), seed);
+ return seed;
+ }
+};
+
+struct IteratorItem {
+ std::unique_ptr<ColumnIterator> iterator;
+ // for holding the reference of segment to avoid use after release
+ SegmentSharedPtr segment;
+};
+
+Status RowIdStorageReader::read_by_rowids(const PMultiGetRequest& request,
Review Comment:
warning: function 'read_by_rowids' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
Status RowIdStorageReader::read_by_rowids(const PMultiGetRequest& request,
^
```
<details>
<summary>Additional context</summary>
**be/src/exec/rowid_fetcher.cpp:327:** 149 lines including whitespace and comments (threshold 80)
```cpp
Status RowIdStorageReader::read_by_rowids(const PMultiGetRequest& request,
^
```
</details>
##########
be/src/service/internal_service.h:
##########
@@ -17,6 +17,7 @@
#pragma once
+#include <brpc/closure_guard.h>
Review Comment:
warning: 'brpc/closure_guard.h' file not found [clang-diagnostic-error]
```cpp
#include <brpc/closure_guard.h>
^
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [cloud](topn-opt) support topn 2 phase read in cloud mode [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31548:
URL: https://github.com/apache/doris/pull/31548#issuecomment-1970397150
<details>
<summary>ClickBench: <b>Total hot run time: 32.42 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b81df8fe71b4b95817d8f4b11cc2b26102492c9a, data reload: false
query1 0.03 0.03 0.03
query2 0.06 0.02 0.02
query3 0.24 0.06 0.05
query4 1.64 0.11 0.10
query5 0.53 0.51 0.52
query6 1.20 0.68 0.68
query7 0.02 0.01 0.01
query8 0.03 0.02 0.02
query9 0.56 0.51 0.52
query10 0.55 0.56 0.57
query11 0.12 0.09 0.09
query12 0.13 0.10 0.10
query13 0.62 0.62 0.62
query14 0.79 0.80 0.81
query15 0.82 0.82 0.82
query16 0.38 0.39 0.37
query17 1.00 1.01 0.98
query18 0.26 0.26 0.26
query19 1.81 1.70 1.71
query20 0.02 0.01 0.01
query21 15.40 0.62 0.62
query22 3.22 4.46 2.37
query23 17.42 1.12 1.06
query24 2.31 0.54 0.50
query25 0.22 0.10 0.04
query26 0.17 0.14 0.13
query27 0.03 0.03 0.03
query28 11.85 0.84 0.83
query29 12.56 3.46 3.26
query30 0.54 0.53 0.57
query31 2.79 0.37 0.36
query32 3.30 0.47 0.48
query33 3.18 3.24 3.21
query34 15.36 4.77 4.75
query35 4.75 4.71 4.74
query36 1.14 1.08 1.06
query37 0.07 0.05 0.05
query38 0.04 0.02 0.02
query39 0.03 0.01 0.02
query40 0.16 0.13 0.13
query41 0.07 0.02 0.01
query42 0.03 0.01 0.02
query43 0.02 0.02 0.02
Total cold run time: 105.47 s
Total hot run time: 32.42 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [cloud](topn-opt) support topn 2 phase read in cloud mode [doris]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on code in PR #31548:
URL: https://github.com/apache/doris/pull/31548#discussion_r1506911707
##########
be/src/runtime/descriptors.h:
##########
@@ -406,6 +407,7 @@ class TupleDescriptor {
friend class SchemaScanner;
friend class OlapTableSchemaParam;
friend class PInternalServiceImpl;
+ friend class RowIdStorageReader;
Review Comment:
used for access constructor in RowIdStorageReader
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [cloud](topn-opt) support topn 2 phase read in cloud mode [doris]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on PR #31548:
URL: https://github.com/apache/doris/pull/31548#issuecomment-1968825870
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [cloud](topn-opt) support topn 2 phase read in cloud mode [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31548:
URL: https://github.com/apache/doris/pull/31548#issuecomment-1970793275
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [cloud](topn-opt) support topn 2 phase read in cloud mode [doris]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on code in PR #31548:
URL: https://github.com/apache/doris/pull/31548#discussion_r1506911068
##########
be/src/cloud/cloud_internal_service.cpp:
##########
@@ -17,11 +17,39 @@
#include "cloud/cloud_internal_service.h"
+#include <brpc/closure_guard.h>
+
+#include "cloud/cloud_storage_engine.h"
+#include "common/signal_handler.h"
+#include "exec/rowid_fetcher.h"
+
namespace doris {
CloudInternalServiceImpl::CloudInternalServiceImpl(CloudStorageEngine& engine, ExecEnv* exec_env)
: PInternalService(exec_env), _engine(engine) {}
CloudInternalServiceImpl::~CloudInternalServiceImpl() = default;
+void CloudInternalServiceImpl::multiget_data(google::protobuf::RpcController* controller,
+ const PMultiGetRequest* request,
+ PMultiGetResponse* response,
+ google::protobuf::Closure* done) {
+ bool ret = _light_work_pool.try_offer([request, response, done, this]() {
Review Comment:
no, they are different interface both inherit from PInternalService
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [cloud](topn-opt) support topn 2 phase read in cloud mode [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31548:
URL: https://github.com/apache/doris/pull/31548#issuecomment-1970384765
<details>
<summary>TPC-H: <b>Total hot run time: 37837 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b81df8fe71b4b95817d8f4b11cc2b26102492c9a, data reload: false
------ Round 1 ----------------------------------
q1 17701 4044 4052 4044
q2 2050 146 137 137
q3 10569 952 943 943
q4 4660 925 943 925
q5 7622 2858 3021 2858
q6 180 122 123 122
q7 1258 764 745 745
q8 9300 2036 2033 2033
q9 7756 6326 6295 6295
q10 8204 2495 2493 2493
q11 423 219 198 198
q12 819 314 321 314
q13 17967 3252 3242 3242
q14 278 257 252 252
q15 558 494 488 488
q16 490 404 408 404
q17 946 886 868 868
q18 6792 5884 6152 5884
q19 1583 1516 1513 1513
q20 556 287 268 268
q21 6302 3528 3600 3528
q22 796 291 283 283
Total cold run time: 106810 ms
Total hot run time: 37837 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4013 4008 4063 4008
q2 308 209 225 209
q3 2997 2935 2918 2918
q4 1850 1820 1810 1810
q5 5149 5079 5112 5079
q6 201 116 117 116
q7 2179 1722 1724 1722
q8 3133 3207 3212 3207
q9 8250 8191 8204 8191
q10 6072 3640 3648 3640
q11 531 412 435 412
q12 672 535 548 535
q13 9567 3044 3042 3042
q14 268 257 256 256
q15 541 491 493 491
q16 506 481 456 456
q17 1696 1682 1668 1668
q18 7916 7690 7400 7400
q19 2194 1651 1636 1636
q20 2104 1923 1891 1891
q21 4773 4600 4568 4568
q22 549 459 439 439
Total cold run time: 65469 ms
Total hot run time: 53694 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [cloud](topn-opt) support topn 2 phase read in cloud mode [doris]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on PR #31548:
URL: https://github.com/apache/doris/pull/31548#issuecomment-1970318485
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [cloud](topn-opt) support topn 2 phase read in cloud mode [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31548:
URL: https://github.com/apache/doris/pull/31548#issuecomment-1970793351
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [cloud](topn-opt) support topn 2 phase read in cloud mode [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31548:
URL: https://github.com/apache/doris/pull/31548#issuecomment-1968987395
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit 5f662e4ccaef51e09880109e45b5b1dab6a9ad5e with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 18.1 seconds inserted 10000000 Rows, about 552K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [cloud](topn-opt) support topn 2 phase read in cloud mode [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring merged PR #31548:
URL: https://github.com/apache/doris/pull/31548
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [cloud](topn-opt) support topn 2 phase read in cloud mode [doris]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on code in PR #31548:
URL: https://github.com/apache/doris/pull/31548#discussion_r1506911707
##########
be/src/runtime/descriptors.h:
##########
@@ -406,6 +407,7 @@ class TupleDescriptor {
friend class SchemaScanner;
friend class OlapTableSchemaParam;
friend class PInternalServiceImpl;
+ friend class RowIdStorageReader;
Review Comment:
used for access in RowIdStorageReader
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [cloud](topn-opt) support topn 2 phase read in cloud mode [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31548:
URL: https://github.com/apache/doris/pull/31548#issuecomment-1970399574
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit b81df8fe71b4b95817d8f4b11cc2b26102492c9a with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 60 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select: 19.3 seconds inserted 10000000 Rows, about 518K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [cloud](topn-opt) support topn 2 phase read in cloud mode [doris]
Posted by "xiaokang (via GitHub)" <gi...@apache.org>.
xiaokang commented on code in PR #31548:
URL: https://github.com/apache/doris/pull/31548#discussion_r1506837883
##########
be/src/exec/rowid_fetcher.cpp:
##########
@@ -275,4 +284,197 @@ Status RowIDFetcher::fetch(const vectorized::ColumnPtr& column_row_ids,
return Status::OK();
}
+template <typename Func>
+auto scope_timer_run(Func fn, int64_t* cost) -> decltype(fn()) {
+ MonotonicStopWatch watch;
+ watch.start();
+ auto res = fn();
+ *cost += watch.elapsed_time() / 1000 / 1000;
+ return res;
+}
+
+struct IteratorKey {
+ int64_t tablet_id;
+ RowsetId rowset_id;
+ uint64_t segment_id;
+ int slot_id;
+
+ // unordered map std::equal_to
+ bool operator==(const IteratorKey& rhs) const {
+ return tablet_id == rhs.tablet_id && rowset_id == rhs.rowset_id &&
+ segment_id == rhs.segment_id && slot_id == rhs.slot_id;
+ }
+};
+
+struct HashOfIteratorKey {
+ size_t operator()(const IteratorKey& key) const {
+ size_t seed = 0;
+ seed = HashUtil::hash64(&key.tablet_id, sizeof(key.tablet_id), seed);
+ seed = HashUtil::hash64(&key.rowset_id.hi, sizeof(key.rowset_id.hi), seed);
+ seed = HashUtil::hash64(&key.rowset_id.mi, sizeof(key.rowset_id.mi), seed);
+ seed = HashUtil::hash64(&key.rowset_id.lo, sizeof(key.rowset_id.lo), seed);
+ seed = HashUtil::hash64(&key.segment_id, sizeof(key.segment_id), seed);
+ seed = HashUtil::hash64(&key.slot_id, sizeof(key.slot_id), seed);
+ return seed;
+ }
+};
+
+struct IteratorItem {
+ std::unique_ptr<ColumnIterator> iterator;
+ // for holding the reference of segment to avoid use after release
+ SegmentSharedPtr segment;
+};
+
+Status RowIdStorageReader::read_by_rowids(const PMultiGetRequest& request,
+ PMultiGetResponse* response) {
+ // read from storage engine row id by row id
+ OlapReaderStatistics stats;
+ vectorized::Block result_block;
+ int64_t acquire_tablet_ms = 0;
+ int64_t acquire_rowsets_ms = 0;
+ int64_t acquire_segments_ms = 0;
+ int64_t lookup_row_data_ms = 0;
+
+ // init desc
+ TupleDescriptor desc(request.desc());
+ std::vector<SlotDescriptor> slots;
+ slots.reserve(request.slots().size());
+ for (const auto& pslot : request.slots()) {
+ slots.push_back(SlotDescriptor(pslot));
+ desc.add_slot(&slots.back());
+ }
+
+ // init read schema
+ TabletSchema full_read_schema;
+ for (const ColumnPB& column_pb : request.column_desc()) {
+ full_read_schema.append_column(TabletColumn(column_pb));
+ }
+
+ std::unordered_map<IteratorKey, IteratorItem, HashOfIteratorKey> iterator_map;
+ // read row by row
+ for (size_t i = 0; i < request.row_locs_size(); ++i) {
+ const auto& row_loc = request.row_locs(i);
+ MonotonicStopWatch watch;
+ watch.start();
+ BaseTabletSPtr tablet = scope_timer_run(
+ [&]() {
+ if (!config::is_cloud_mode()) {
+ auto tablet =
+ static_cast<StorageEngine*>(_engine)->tablet_manager()->get_tablet(
+ row_loc.tablet_id(), true /*include deleted*/);
+ return std::dynamic_pointer_cast<BaseTablet>(tablet);
+ } else {
+ auto res =
+ static_cast<CloudStorageEngine*>(_engine)->tablet_mgr().get_tablet(
+ row_loc.tablet_id(), true /*include deleted*/);
+ return std::dynamic_pointer_cast<BaseTablet>(res.value());
+ }
+ },
+ &acquire_tablet_ms);
+ RowsetId rowset_id;
+ rowset_id.init(row_loc.rowset_id());
+ if (!tablet) {
+ continue;
+ }
+ // We ensured it's rowset is not released when init Tablet reader param, rowset->update_delayed_expired_timestamp();
+ BetaRowsetSharedPtr rowset = std::static_pointer_cast<BetaRowset>(scope_timer_run(
+ [&]() { return _engine->get_quering_rowset(rowset_id); }, &acquire_rowsets_ms));
+ if (!rowset) {
+ LOG(INFO) << "no such rowset " << rowset_id;
+ continue;
+ }
+ size_t row_size = 0;
+ Defer _defer([&]() {
+ LOG_EVERY_N(INFO, 100)
+ << "multiget_data single_row, cost(us):" << watch.elapsed_time() / 1000
+ << ", row_size:" << row_size;
+ *response->add_row_locs() = row_loc;
+ });
+ SegmentCacheHandle segment_cache;
+ RETURN_IF_ERROR(scope_timer_run(
+ [&]() {
+ return SegmentLoader::instance()->load_segments(rowset, &segment_cache, true);
Review Comment:
Is SegmentLoader the same for both cloud and local mode?
##########
be/src/runtime/descriptors.h:
##########
@@ -406,6 +407,7 @@ class TupleDescriptor {
friend class SchemaScanner;
friend class OlapTableSchemaParam;
friend class PInternalServiceImpl;
+ friend class RowIdStorageReader;
Review Comment:
not used.
##########
be/src/exec/rowid_fetcher.cpp:
##########
@@ -275,4 +284,197 @@ Status RowIDFetcher::fetch(const vectorized::ColumnPtr& column_row_ids,
return Status::OK();
}
+template <typename Func>
+auto scope_timer_run(Func fn, int64_t* cost) -> decltype(fn()) {
+ MonotonicStopWatch watch;
+ watch.start();
+ auto res = fn();
+ *cost += watch.elapsed_time() / 1000 / 1000;
+ return res;
+}
+
+struct IteratorKey {
+ int64_t tablet_id;
+ RowsetId rowset_id;
+ uint64_t segment_id;
+ int slot_id;
+
+ // unordered map std::equal_to
+ bool operator==(const IteratorKey& rhs) const {
+ return tablet_id == rhs.tablet_id && rowset_id == rhs.rowset_id &&
+ segment_id == rhs.segment_id && slot_id == rhs.slot_id;
+ }
+};
+
+struct HashOfIteratorKey {
+ size_t operator()(const IteratorKey& key) const {
+ size_t seed = 0;
+ seed = HashUtil::hash64(&key.tablet_id, sizeof(key.tablet_id), seed);
+ seed = HashUtil::hash64(&key.rowset_id.hi, sizeof(key.rowset_id.hi), seed);
+ seed = HashUtil::hash64(&key.rowset_id.mi, sizeof(key.rowset_id.mi), seed);
+ seed = HashUtil::hash64(&key.rowset_id.lo, sizeof(key.rowset_id.lo), seed);
+ seed = HashUtil::hash64(&key.segment_id, sizeof(key.segment_id), seed);
+ seed = HashUtil::hash64(&key.slot_id, sizeof(key.slot_id), seed);
+ return seed;
+ }
+};
+
+struct IteratorItem {
+ std::unique_ptr<ColumnIterator> iterator;
+ // for holding the reference of segment to avoid use after release
+ SegmentSharedPtr segment;
+};
+
+Status RowIdStorageReader::read_by_rowids(const PMultiGetRequest& request,
+ PMultiGetResponse* response) {
+ // read from storage engine row id by row id
+ OlapReaderStatistics stats;
+ vectorized::Block result_block;
+ int64_t acquire_tablet_ms = 0;
+ int64_t acquire_rowsets_ms = 0;
+ int64_t acquire_segments_ms = 0;
+ int64_t lookup_row_data_ms = 0;
+
+ // init desc
+ TupleDescriptor desc(request.desc());
+ std::vector<SlotDescriptor> slots;
+ slots.reserve(request.slots().size());
+ for (const auto& pslot : request.slots()) {
+ slots.push_back(SlotDescriptor(pslot));
+ desc.add_slot(&slots.back());
+ }
+
+ // init read schema
+ TabletSchema full_read_schema;
+ for (const ColumnPB& column_pb : request.column_desc()) {
+ full_read_schema.append_column(TabletColumn(column_pb));
+ }
+
+ std::unordered_map<IteratorKey, IteratorItem, HashOfIteratorKey> iterator_map;
+ // read row by row
+ for (size_t i = 0; i < request.row_locs_size(); ++i) {
+ const auto& row_loc = request.row_locs(i);
+ MonotonicStopWatch watch;
+ watch.start();
+ BaseTabletSPtr tablet = scope_timer_run(
+ [&]() {
+ if (!config::is_cloud_mode()) {
Review Comment:
A consistent interface should be used here for cloud and local mode, instead of if else.
##########
be/src/exec/rowid_fetcher.cpp:
##########
@@ -275,4 +284,197 @@ Status RowIDFetcher::fetch(const vectorized::ColumnPtr& column_row_ids,
return Status::OK();
}
+template <typename Func>
+auto scope_timer_run(Func fn, int64_t* cost) -> decltype(fn()) {
+ MonotonicStopWatch watch;
+ watch.start();
+ auto res = fn();
+ *cost += watch.elapsed_time() / 1000 / 1000;
+ return res;
+}
+
+struct IteratorKey {
+ int64_t tablet_id;
+ RowsetId rowset_id;
+ uint64_t segment_id;
+ int slot_id;
+
+ // unordered map std::equal_to
+ bool operator==(const IteratorKey& rhs) const {
+ return tablet_id == rhs.tablet_id && rowset_id == rhs.rowset_id &&
+ segment_id == rhs.segment_id && slot_id == rhs.slot_id;
+ }
+};
+
+struct HashOfIteratorKey {
+ size_t operator()(const IteratorKey& key) const {
+ size_t seed = 0;
+ seed = HashUtil::hash64(&key.tablet_id, sizeof(key.tablet_id), seed);
+ seed = HashUtil::hash64(&key.rowset_id.hi, sizeof(key.rowset_id.hi), seed);
+ seed = HashUtil::hash64(&key.rowset_id.mi, sizeof(key.rowset_id.mi), seed);
+ seed = HashUtil::hash64(&key.rowset_id.lo, sizeof(key.rowset_id.lo), seed);
+ seed = HashUtil::hash64(&key.segment_id, sizeof(key.segment_id), seed);
+ seed = HashUtil::hash64(&key.slot_id, sizeof(key.slot_id), seed);
+ return seed;
+ }
+};
+
+struct IteratorItem {
+ std::unique_ptr<ColumnIterator> iterator;
+ // for holding the reference of segment to avoid use after release
+ SegmentSharedPtr segment;
+};
+
+Status RowIdStorageReader::read_by_rowids(const PMultiGetRequest& request,
+ PMultiGetResponse* response) {
+ // read from storage engine row id by row id
+ OlapReaderStatistics stats;
+ vectorized::Block result_block;
+ int64_t acquire_tablet_ms = 0;
+ int64_t acquire_rowsets_ms = 0;
+ int64_t acquire_segments_ms = 0;
+ int64_t lookup_row_data_ms = 0;
+
+ // init desc
+ TupleDescriptor desc(request.desc());
+ std::vector<SlotDescriptor> slots;
+ slots.reserve(request.slots().size());
+ for (const auto& pslot : request.slots()) {
+ slots.push_back(SlotDescriptor(pslot));
+ desc.add_slot(&slots.back());
+ }
+
+ // init read schema
+ TabletSchema full_read_schema;
+ for (const ColumnPB& column_pb : request.column_desc()) {
+ full_read_schema.append_column(TabletColumn(column_pb));
+ }
+
+ std::unordered_map<IteratorKey, IteratorItem, HashOfIteratorKey> iterator_map;
+ // read row by row
+ for (size_t i = 0; i < request.row_locs_size(); ++i) {
+ const auto& row_loc = request.row_locs(i);
+ MonotonicStopWatch watch;
+ watch.start();
+ BaseTabletSPtr tablet = scope_timer_run(
+ [&]() {
+ if (!config::is_cloud_mode()) {
Review Comment:
eg. the same tablet_manager() or get_tablet() interface in BaseStorageEngine.
##########
be/src/cloud/cloud_internal_service.cpp:
##########
@@ -17,11 +17,39 @@
#include "cloud/cloud_internal_service.h"
+#include <brpc/closure_guard.h>
+
+#include "cloud/cloud_storage_engine.h"
+#include "common/signal_handler.h"
+#include "exec/rowid_fetcher.h"
+
namespace doris {
CloudInternalServiceImpl::CloudInternalServiceImpl(CloudStorageEngine& engine, ExecEnv* exec_env)
: PInternalService(exec_env), _engine(engine) {}
CloudInternalServiceImpl::~CloudInternalServiceImpl() = default;
+void CloudInternalServiceImpl::multiget_data(google::protobuf::RpcController* controller,
+ const PMultiGetRequest* request,
+ PMultiGetResponse* response,
+ google::protobuf::Closure* done) {
+ bool ret = _light_work_pool.try_offer([request, response, done, this]() {
Review Comment:
It's the same as PInternalServiceImpl::multiget_data. Can you call it?
##########
be/src/olap/olap_server.cpp:
##########
@@ -129,6 +129,12 @@ Status StorageEngine::start_bg_threads() {
&_unused_rowset_monitor_thread));
LOG(INFO) << "unused rowset monitor thread started";
+ RETURN_IF_ERROR(Thread::create(
Review Comment:
There is no such thread before. Is it special for cloud mode?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [cloud](topn-opt) support topn 2 phase read in cloud mode [doris]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on PR #31548:
URL: https://github.com/apache/doris/pull/31548#issuecomment-1968827798
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [cloud](topn-opt) support topn 2 phase read in cloud mode [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31548:
URL: https://github.com/apache/doris/pull/31548#issuecomment-1968972485
<details>
<summary>TPC-DS: <b>Total hot run time: 169918 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5f662e4ccaef51e09880109e45b5b1dab6a9ad5e, data reload: false
query1 941 346 338 338
query2 6523 1740 1806 1740
query3 6699 219 214 214
query4 23420 20635 20629 20629
query5 4256 365 360 360
query6 281 173 170 170
query7 4603 293 301 293
query8 247 193 200 193
query9 8416 2245 2250 2245
query10 423 217 214 214
query11 14695 14107 14125 14107
query12 138 84 91 84
query13 1671 420 432 420
query14 8860 6597 6662 6597
query15 219 179 178 178
query16 7086 268 267 267
query17 1016 525 524 524
query18 1897 268 261 261
query19 186 146 156 146
query20 87 80 82 80
query21 195 121 123 121
query22 4811 4631 4418 4418
query23 30485 29858 29951 29858
query24 11903 3084 3079 3079
query25 643 359 355 355
query26 1805 152 163 152
query27 2957 312 322 312
query28 6963 1822 1823 1822
query29 1230 569 560 560
query30 289 136 141 136
query31 855 694 691 691
query32 95 61 61 61
query33 709 228 223 223
query34 1063 498 495 495
query35 867 756 773 756
query36 965 921 913 913
query37 253 62 64 62
query38 3145 2998 3023 2998
query39 1333 1260 1248 1248
query40 285 100 99 99
query41 39 38 35 35
query42 104 98 98 98
query43 473 438 430 430
query44 1079 691 713 691
query45 189 178 171 171
query46 1043 806 793 793
query47 1544 1418 1423 1418
query48 427 348 348 348
query49 1181 306 298 298
query50 776 377 383 377
query51 4410 4316 4288 4288
query52 100 97 89 89
query53 341 268 267 267
query54 298 225 224 224
query55 88 87 82 82
query56 217 200 201 200
query57 944 899 918 899
query58 204 191 197 191
query59 2157 2166 2147 2147
query60 253 217 237 217
query61 84 84 81 81
query62 596 358 325 325
query63 281 254 261 254
query64 6494 3017 3135 3017
query65 3260 3239 3214 3214
query66 1432 342 319 319
query67 14493 14108 14190 14108
query68 5013 555 565 555
query69 543 381 397 381
query70 1226 1213 1272 1213
query71 397 252 262 252
query72 6168 2793 2687 2687
query73 712 320 321 320
query74 6392 5976 5937 5937
query75 3083 2451 2417 2417
query76 2779 1069 1134 1069
query77 354 247 243 243
query78 9063 8688 8515 8515
query79 1462 524 517 517
query80 1074 353 354 353
query81 480 202 193 193
query82 1192 85 88 85
query83 231 126 123 123
query84 277 80 76 76
query85 1327 379 347 347
query86 377 301 305 301
query87 3293 3100 3098 3098
query88 2729 2313 2313 2313
query89 383 339 316 316
query90 1862 167 167 167
query91 166 120 147 120
query92 56 47 52 47
query93 1230 510 513 510
query94 1200 181 178 178
query95 438 345 343 343
query96 570 263 260 260
query97 4259 4133 4136 4133
query98 214 210 192 192
query99 1048 657 683 657
Total cold run time: 264620 ms
Total hot run time: 169918 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [cloud](topn-opt) support topn 2 phase read in cloud mode [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31548:
URL: https://github.com/apache/doris/pull/31548#issuecomment-1968953469
<details>
<summary>TPC-H: <b>Total hot run time: 38064 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5f662e4ccaef51e09880109e45b5b1dab6a9ad5e, data reload: false
------ Round 1 ----------------------------------
q1 17723 4028 4042 4028
q2 2059 153 135 135
q3 10562 1033 950 950
q4 4662 965 956 956
q5 7625 2872 3003 2872
q6 182 125 126 125
q7 1246 775 765 765
q8 9316 2042 2044 2042
q9 7349 6334 6341 6334
q10 8205 2501 2505 2501
q11 422 217 205 205
q12 792 320 312 312
q13 17944 3230 3215 3215
q14 279 258 257 257
q15 542 492 505 492
q16 491 415 428 415
q17 960 885 855 855
q18 6651 6117 5926 5926
q19 1570 1518 1521 1518
q20 548 266 271 266
q21 7024 3612 3621 3612
q22 814 283 291 283
Total cold run time: 106966 ms
Total hot run time: 38064 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4014 4034 3996 3996
q2 304 214 221 214
q3 2978 2926 2952 2926
q4 1827 1827 1819 1819
q5 5132 5118 5119 5118
q6 203 120 119 119
q7 2141 1771 1693 1693
q8 3136 3198 3223 3198
q9 8285 8271 8229 8229
q10 6121 3682 3677 3677
q11 512 436 438 436
q12 694 515 531 515
q13 9350 3064 3063 3063
q14 285 256 250 250
q15 533 493 496 493
q16 488 463 448 448
q17 1725 1702 1688 1688
q18 7741 7531 7401 7401
q19 1685 1662 1645 1645
q20 2117 1917 1902 1902
q21 4760 4662 4577 4577
q22 518 451 449 449
Total cold run time: 64549 ms
Total hot run time: 53856 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [cloud](topn-opt) support topn 2 phase read in cloud mode [doris]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on code in PR #31548:
URL: https://github.com/apache/doris/pull/31548#discussion_r1506940049
##########
be/src/cloud/cloud_internal_service.cpp:
##########
@@ -17,11 +17,39 @@
#include "cloud/cloud_internal_service.h"
+#include <brpc/closure_guard.h>
+
+#include "cloud/cloud_storage_engine.h"
+#include "common/signal_handler.h"
+#include "exec/rowid_fetcher.h"
+
namespace doris {
CloudInternalServiceImpl::CloudInternalServiceImpl(CloudStorageEngine& engine, ExecEnv* exec_env)
: PInternalService(exec_env), _engine(engine) {}
CloudInternalServiceImpl::~CloudInternalServiceImpl() = default;
+void CloudInternalServiceImpl::multiget_data(google::protobuf::RpcController* controller,
+ const PMultiGetRequest* request,
+ PMultiGetResponse* response,
+ google::protobuf::Closure* done) {
+ bool ret = _light_work_pool.try_offer([request, response, done, this]() {
Review Comment:
done
##########
be/src/exec/rowid_fetcher.cpp:
##########
@@ -275,4 +284,197 @@ Status RowIDFetcher::fetch(const vectorized::ColumnPtr& column_row_ids,
return Status::OK();
}
+template <typename Func>
+auto scope_timer_run(Func fn, int64_t* cost) -> decltype(fn()) {
+ MonotonicStopWatch watch;
+ watch.start();
+ auto res = fn();
+ *cost += watch.elapsed_time() / 1000 / 1000;
+ return res;
+}
+
+struct IteratorKey {
+ int64_t tablet_id;
+ RowsetId rowset_id;
+ uint64_t segment_id;
+ int slot_id;
+
+ // unordered map std::equal_to
+ bool operator==(const IteratorKey& rhs) const {
+ return tablet_id == rhs.tablet_id && rowset_id == rhs.rowset_id &&
+ segment_id == rhs.segment_id && slot_id == rhs.slot_id;
+ }
+};
+
+struct HashOfIteratorKey {
+ size_t operator()(const IteratorKey& key) const {
+ size_t seed = 0;
+ seed = HashUtil::hash64(&key.tablet_id, sizeof(key.tablet_id), seed);
+ seed = HashUtil::hash64(&key.rowset_id.hi, sizeof(key.rowset_id.hi), seed);
+ seed = HashUtil::hash64(&key.rowset_id.mi, sizeof(key.rowset_id.mi), seed);
+ seed = HashUtil::hash64(&key.rowset_id.lo, sizeof(key.rowset_id.lo), seed);
+ seed = HashUtil::hash64(&key.segment_id, sizeof(key.segment_id), seed);
+ seed = HashUtil::hash64(&key.slot_id, sizeof(key.slot_id), seed);
+ return seed;
+ }
+};
+
+struct IteratorItem {
+ std::unique_ptr<ColumnIterator> iterator;
+ // for holding the reference of segment to avoid use after release
+ SegmentSharedPtr segment;
+};
+
+Status RowIdStorageReader::read_by_rowids(const PMultiGetRequest& request,
+ PMultiGetResponse* response) {
+ // read from storage engine row id by row id
+ OlapReaderStatistics stats;
+ vectorized::Block result_block;
+ int64_t acquire_tablet_ms = 0;
+ int64_t acquire_rowsets_ms = 0;
+ int64_t acquire_segments_ms = 0;
+ int64_t lookup_row_data_ms = 0;
+
+ // init desc
+ TupleDescriptor desc(request.desc());
+ std::vector<SlotDescriptor> slots;
+ slots.reserve(request.slots().size());
+ for (const auto& pslot : request.slots()) {
+ slots.push_back(SlotDescriptor(pslot));
+ desc.add_slot(&slots.back());
+ }
+
+ // init read schema
+ TabletSchema full_read_schema;
+ for (const ColumnPB& column_pb : request.column_desc()) {
+ full_read_schema.append_column(TabletColumn(column_pb));
+ }
+
+ std::unordered_map<IteratorKey, IteratorItem, HashOfIteratorKey> iterator_map;
+ // read row by row
+ for (size_t i = 0; i < request.row_locs_size(); ++i) {
+ const auto& row_loc = request.row_locs(i);
+ MonotonicStopWatch watch;
+ watch.start();
+ BaseTabletSPtr tablet = scope_timer_run(
+ [&]() {
+ if (!config::is_cloud_mode()) {
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [cloud](topn-opt) support topn 2 phase read in cloud mode [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31548:
URL: https://github.com/apache/doris/pull/31548#issuecomment-1970393164
<details>
<summary>TPC-DS: <b>Total hot run time: 168404 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b81df8fe71b4b95817d8f4b11cc2b26102492c9a, data reload: false
query1 928 339 332 332
query2 6537 1645 1621 1621
query3 6700 215 203 203
query4 23653 20488 20485 20485
query5 4261 363 351 351
query6 260 164 166 164
query7 4606 292 280 280
query8 265 205 187 187
query9 8471 2250 2258 2250
query10 422 212 209 209
query11 14520 14161 14037 14037
query12 137 81 82 81
query13 1634 415 423 415
query14 8258 6388 6284 6284
query15 205 172 173 172
query16 7078 267 250 250
query17 1042 524 490 490
query18 1897 267 269 267
query19 186 145 143 143
query20 85 81 81 81
query21 196 119 129 119
query22 4751 4667 4590 4590
query23 30540 29924 29875 29875
query24 12379 3061 3090 3061
query25 652 348 354 348
query26 1914 156 165 156
query27 3056 301 320 301
query28 7080 1807 1805 1805
query29 1253 559 559 559
query30 283 135 138 135
query31 893 692 690 690
query32 99 59 61 59
query33 708 225 219 219
query34 1061 477 495 477
query35 845 741 744 741
query36 937 874 798 798
query37 187 59 63 59
query38 3078 2982 2992 2982
query39 1311 1263 1239 1239
query40 283 98 98 98
query41 38 36 36 36
query42 104 92 96 92
query43 451 436 429 429
query44 1059 687 701 687
query45 192 177 169 169
query46 1042 797 763 763
query47 1511 1367 1417 1367
query48 414 332 332 332
query49 1185 290 285 285
query50 774 378 364 364
query51 4352 4292 4248 4248
query52 102 92 89 89
query53 325 266 263 263
query54 288 219 219 219
query55 86 81 87 81
query56 215 200 199 199
query57 933 873 884 873
query58 214 197 189 189
query59 2181 2008 2043 2008
query60 236 209 208 208
query61 85 85 87 85
query62 595 364 353 353
query63 282 258 253 253
query64 6500 3035 3117 3035
query65 3243 3257 3222 3222
query66 1436 317 318 317
query67 14258 14199 14079 14079
query68 4978 537 546 537
query69 518 372 362 362
query70 1257 1222 1190 1190
query71 319 237 242 237
query72 6262 2803 2669 2669
query73 696 311 320 311
query74 6291 5950 5881 5881
query75 3035 2436 2416 2416
query76 2753 1041 1155 1041
query77 336 239 237 237
query78 9006 8586 8453 8453
query79 1208 516 520 516
query80 1010 349 344 344
query81 489 196 195 195
query82 960 84 84 84
query83 225 120 118 118
query84 282 76 79 76
query85 1212 352 344 344
query86 387 297 314 297
query87 3304 3121 3090 3090
query88 2673 2268 2237 2237
query89 408 324 324 324
query90 1883 168 168 168
query91 163 120 123 120
query92 53 46 47 46
query93 1219 511 507 507
query94 1258 179 171 171
query95 439 337 338 337
query96 578 259 260 259
query97 4258 4122 4174 4122
query98 212 206 194 194
query99 1055 667 663 663
Total cold run time: 263479 ms
Total hot run time: 168404 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [cloud](topn-opt) support topn 2 phase read in cloud mode [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31548:
URL: https://github.com/apache/doris/pull/31548#issuecomment-1968981812
<details>
<summary>ClickBench: <b>Total hot run time: 32.09 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5f662e4ccaef51e09880109e45b5b1dab6a9ad5e, data reload: false
query1 0.03 0.03 0.03
query2 0.05 0.02 0.02
query3 0.23 0.06 0.06
query4 1.68 0.10 0.10
query5 0.52 0.51 0.51
query6 1.34 0.67 0.66
query7 0.02 0.02 0.01
query8 0.04 0.02 0.02
query9 0.56 0.50 0.51
query10 0.57 0.56 0.58
query11 0.12 0.10 0.09
query12 0.11 0.10 0.10
query13 0.62 0.61 0.61
query14 0.81 0.81 0.79
query15 0.84 0.82 0.82
query16 0.37 0.36 0.36
query17 0.97 0.98 1.00
query18 0.26 0.26 0.25
query19 1.74 1.68 1.73
query20 0.02 0.01 0.01
query21 15.42 0.62 0.67
query22 3.32 3.91 2.31
query23 17.48 1.03 0.94
query24 1.96 0.59 0.56
query25 0.48 0.11 0.07
query26 0.16 0.12 0.13
query27 0.06 0.05 0.05
query28 12.03 0.85 0.84
query29 12.56 3.07 3.12
query30 0.62 0.53 0.52
query31 2.79 0.35 0.36
query32 3.36 0.49 0.49
query33 3.19 3.21 3.19
query34 15.35 4.75 4.76
query35 4.78 4.75 4.72
query36 1.14 1.08 1.06
query37 0.06 0.05 0.06
query38 0.04 0.03 0.03
query39 0.01 0.01 0.02
query40 0.16 0.14 0.13
query41 0.07 0.01 0.02
query42 0.02 0.02 0.02
query43 0.03 0.02 0.03
Total cold run time: 105.99 s
Total hot run time: 32.09 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [cloud](topn-opt) support topn 2 phase read in cloud mode [doris]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on code in PR #31548:
URL: https://github.com/apache/doris/pull/31548#discussion_r1506911068
##########
be/src/cloud/cloud_internal_service.cpp:
##########
@@ -17,11 +17,39 @@
#include "cloud/cloud_internal_service.h"
+#include <brpc/closure_guard.h>
+
+#include "cloud/cloud_storage_engine.h"
+#include "common/signal_handler.h"
+#include "exec/rowid_fetcher.h"
+
namespace doris {
CloudInternalServiceImpl::CloudInternalServiceImpl(CloudStorageEngine& engine, ExecEnv* exec_env)
: PInternalService(exec_env), _engine(engine) {}
CloudInternalServiceImpl::~CloudInternalServiceImpl() = default;
+void CloudInternalServiceImpl::multiget_data(google::protobuf::RpcController* controller,
+ const PMultiGetRequest* request,
+ PMultiGetResponse* response,
+ google::protobuf::Closure* done) {
+ bool ret = _light_work_pool.try_offer([request, response, done, this]() {
Review Comment:
no, they are different interface both inherit from PInternalService
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [cloud](topn-opt) support topn 2 phase read in cloud mode [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31548:
URL: https://github.com/apache/doris/pull/31548#issuecomment-1970397750
TeamCity be ut coverage result:
Function Coverage: 35.67% (8546/23957)
Line Coverage: 27.48% (69361/252449)
Region Coverage: 26.63% (35965/135079)
Branch Coverage: 23.44% (18385/78436)
Coverage Report: http://coverage.selectdb-in.cc/coverage/b81df8fe71b4b95817d8f4b11cc2b26102492c9a_b81df8fe71b4b95817d8f4b11cc2b26102492c9a/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [cloud](topn-opt) support topn 2 phase read in cloud mode [doris]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on code in PR #31548:
URL: https://github.com/apache/doris/pull/31548#discussion_r1506911281
##########
be/src/exec/rowid_fetcher.cpp:
##########
@@ -275,4 +284,197 @@ Status RowIDFetcher::fetch(const vectorized::ColumnPtr& column_row_ids,
return Status::OK();
}
+template <typename Func>
+auto scope_timer_run(Func fn, int64_t* cost) -> decltype(fn()) {
+ MonotonicStopWatch watch;
+ watch.start();
+ auto res = fn();
+ *cost += watch.elapsed_time() / 1000 / 1000;
+ return res;
+}
+
+struct IteratorKey {
+ int64_t tablet_id;
+ RowsetId rowset_id;
+ uint64_t segment_id;
+ int slot_id;
+
+ // unordered map std::equal_to
+ bool operator==(const IteratorKey& rhs) const {
+ return tablet_id == rhs.tablet_id && rowset_id == rhs.rowset_id &&
+ segment_id == rhs.segment_id && slot_id == rhs.slot_id;
+ }
+};
+
+struct HashOfIteratorKey {
+ size_t operator()(const IteratorKey& key) const {
+ size_t seed = 0;
+ seed = HashUtil::hash64(&key.tablet_id, sizeof(key.tablet_id), seed);
+ seed = HashUtil::hash64(&key.rowset_id.hi, sizeof(key.rowset_id.hi), seed);
+ seed = HashUtil::hash64(&key.rowset_id.mi, sizeof(key.rowset_id.mi), seed);
+ seed = HashUtil::hash64(&key.rowset_id.lo, sizeof(key.rowset_id.lo), seed);
+ seed = HashUtil::hash64(&key.segment_id, sizeof(key.segment_id), seed);
+ seed = HashUtil::hash64(&key.slot_id, sizeof(key.slot_id), seed);
+ return seed;
+ }
+};
+
+struct IteratorItem {
+ std::unique_ptr<ColumnIterator> iterator;
+ // for holding the reference of segment to avoid use after release
+ SegmentSharedPtr segment;
+};
+
+Status RowIdStorageReader::read_by_rowids(const PMultiGetRequest& request,
+ PMultiGetResponse* response) {
+ // read from storage engine row id by row id
+ OlapReaderStatistics stats;
+ vectorized::Block result_block;
+ int64_t acquire_tablet_ms = 0;
+ int64_t acquire_rowsets_ms = 0;
+ int64_t acquire_segments_ms = 0;
+ int64_t lookup_row_data_ms = 0;
+
+ // init desc
+ TupleDescriptor desc(request.desc());
+ std::vector<SlotDescriptor> slots;
+ slots.reserve(request.slots().size());
+ for (const auto& pslot : request.slots()) {
+ slots.push_back(SlotDescriptor(pslot));
+ desc.add_slot(&slots.back());
+ }
+
+ // init read schema
+ TabletSchema full_read_schema;
+ for (const ColumnPB& column_pb : request.column_desc()) {
+ full_read_schema.append_column(TabletColumn(column_pb));
+ }
+
+ std::unordered_map<IteratorKey, IteratorItem, HashOfIteratorKey> iterator_map;
+ // read row by row
+ for (size_t i = 0; i < request.row_locs_size(); ++i) {
+ const auto& row_loc = request.row_locs(i);
+ MonotonicStopWatch watch;
+ watch.start();
+ BaseTabletSPtr tablet = scope_timer_run(
+ [&]() {
+ if (!config::is_cloud_mode()) {
+ auto tablet =
+ static_cast<StorageEngine*>(_engine)->tablet_manager()->get_tablet(
+ row_loc.tablet_id(), true /*include deleted*/);
+ return std::dynamic_pointer_cast<BaseTablet>(tablet);
+ } else {
+ auto res =
+ static_cast<CloudStorageEngine*>(_engine)->tablet_mgr().get_tablet(
+ row_loc.tablet_id(), true /*include deleted*/);
+ return std::dynamic_pointer_cast<BaseTablet>(res.value());
+ }
+ },
+ &acquire_tablet_ms);
+ RowsetId rowset_id;
+ rowset_id.init(row_loc.rowset_id());
+ if (!tablet) {
+ continue;
+ }
+ // We ensured it's rowset is not released when init Tablet reader param, rowset->update_delayed_expired_timestamp();
+ BetaRowsetSharedPtr rowset = std::static_pointer_cast<BetaRowset>(scope_timer_run(
+ [&]() { return _engine->get_quering_rowset(rowset_id); }, &acquire_rowsets_ms));
+ if (!rowset) {
+ LOG(INFO) << "no such rowset " << rowset_id;
+ continue;
+ }
+ size_t row_size = 0;
+ Defer _defer([&]() {
+ LOG_EVERY_N(INFO, 100)
+ << "multiget_data single_row, cost(us):" << watch.elapsed_time() / 1000
+ << ", row_size:" << row_size;
+ *response->add_row_locs() = row_loc;
+ });
+ SegmentCacheHandle segment_cache;
+ RETURN_IF_ERROR(scope_timer_run(
+ [&]() {
+ return SegmentLoader::instance()->load_segments(rowset, &segment_cache, true);
Review Comment:
they are same
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [cloud](topn-opt) support topn 2 phase read in cloud mode [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31548:
URL: https://github.com/apache/doris/pull/31548#issuecomment-1968823553
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org