You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "liaoxin01 (via GitHub)" <gi...@apache.org> on 2023/06/12 08:39:31 UTC
[GitHub] [doris] liaoxin01 opened a new pull request, #20706: [Enhancement](merge-on-write) parallel calculate delete bitmap when tablet has multi segments
liaoxin01 opened a new pull request, #20706:
URL: https://github.com/apache/doris/pull/20706
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
In order to accelerate the processing time of publish for merge-on-write table，parallel calculate delete bitmap when tablet has multi segments.
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20706: [Enhancement](merge-on-write) parallel calculate delete bitmap when tablet has multi segments
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20706:
URL: https://github.com/apache/doris/pull/20706#issuecomment-1586864883
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] liaoxin01 commented on pull request #20706: [Enhancement](merge-on-write) parallel calculate delete bitmap when tablet has multi segments
Posted by "liaoxin01 (via GitHub)" <gi...@apache.org>.
liaoxin01 commented on PR #20706:
URL: https://github.com/apache/doris/pull/20706#issuecomment-1588545418
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] dataroaring merged pull request #20706: [Enhancement](merge-on-write) parallel calculate delete bitmap when tablet has multi segments
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring merged PR #20706:
URL: https://github.com/apache/doris/pull/20706
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20706: [Enhancement](merge-on-write) parallel calculate delete bitmap when tablet has multi segments
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20706:
URL: https://github.com/apache/doris/pull/20706#issuecomment-1593020582
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20706: [Enhancement](merge-on-write) parallel calculate delete bitmap when tablet has multi segments
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20706:
URL: https://github.com/apache/doris/pull/20706#issuecomment-1588631369
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] liaoxin01 commented on pull request #20706: [Enhancement](merge-on-write) parallel calculate delete bitmap when tablet has multi segments
Posted by "liaoxin01 (via GitHub)" <gi...@apache.org>.
liaoxin01 commented on PR #20706:
URL: https://github.com/apache/doris/pull/20706#issuecomment-1588625759
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20706: [Enhancement](merge-on-write) parallel calculate delete bitmap when tablet has multi segments
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20706:
URL: https://github.com/apache/doris/pull/20706#issuecomment-1588626061
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20706: [Enhancement](merge-on-write) parallel calculate delete bitmap when tablet has multi segments
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20706:
URL: https://github.com/apache/doris/pull/20706#issuecomment-1588547546
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] zhannngchen commented on a diff in pull request #20706: [Enhancement](merge-on-write) parallel calculate delete bitmap when tablet has multi segments
Posted by "zhannngchen (via GitHub)" <gi...@apache.org>.
zhannngchen commented on code in PR #20706:
URL: https://github.com/apache/doris/pull/20706#discussion_r1229183664
##########
be/src/olap/tablet.cpp:
##########
@@ -2703,9 +2704,11 @@ Status Tablet::lookup_row_data(const Slice& encoded_key, const RowLocation& row_
}
// ATTN: caller should hold the meta lock.
-Status Tablet::lookup_row_key(const Slice& encoded_key, bool with_seq_col,
- const RowsetIdUnorderedSet* rowset_ids, RowLocation* row_location,
- uint32_t version, RowsetSharedPtr* rowset) {
+Status Tablet::lookup_row_key(
+ const Slice& encoded_key, bool with_seq_col, const RowsetIdUnorderedSet* rowset_ids,
+ RowLocation* row_location, uint32_t version,
+ std::unordered_map<RowsetId, SegmentCacheHandle, HashOfRowsetId>& segment_caches,
Review Comment:
emm... one more parameter, we need a refactor later
##########
be/src/service/point_query_executor.cpp:
##########
@@ -264,7 +265,7 @@ Status PointQueryExecutor::_lookup_row_key() {
// Get rowlocation and rowset, ctx._rowset_ptr will acquire wrap this ptr
auto rowset_ptr = std::make_unique<RowsetSharedPtr>();
st = (_tablet->lookup_row_key(_row_read_ctxs[i]._primary_key, true, nullptr, &location,
- INT32_MAX /*rethink?*/, rowset_ptr.get()));
+ INT32_MAX /*rethink?*/, segment_caches, rowset_ptr.get()));
Review Comment:
point lookup should not use such a segment cache? it's meaningless
you should allow to pass a nullptr here.
##########
be/src/olap/rowset/segment_v2/segment_writer.cpp:
##########
@@ -363,6 +365,7 @@ Status SegmentWriter::append_block_with_partial_content(const vectorized::Block*
bool has_default = false;
std::vector<bool> use_default_flag;
use_default_flag.reserve(num_rows);
+ std::unordered_map<RowsetId, SegmentCacheHandle, HashOfRowsetId> segment_caches;
Review Comment:
Add a comment here, why do we need such a cache
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20706: [Enhancement](merge-on-write) parallel calculate delete bitmap when tablet has multi segments
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20706:
URL: https://github.com/apache/doris/pull/20706#issuecomment-1593020478
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] liaoxin01 commented on a diff in pull request #20706: [Enhancement](merge-on-write) parallel calculate delete bitmap when tablet has multi segments
Posted by "liaoxin01 (via GitHub)" <gi...@apache.org>.
liaoxin01 commented on code in PR #20706:
URL: https://github.com/apache/doris/pull/20706#discussion_r1229242541
##########
be/src/olap/tablet.cpp:
##########
@@ -2703,9 +2704,11 @@ Status Tablet::lookup_row_data(const Slice& encoded_key, const RowLocation& row_
}
// ATTN: caller should hold the meta lock.
-Status Tablet::lookup_row_key(const Slice& encoded_key, bool with_seq_col,
- const RowsetIdUnorderedSet* rowset_ids, RowLocation* row_location,
- uint32_t version, RowsetSharedPtr* rowset) {
+Status Tablet::lookup_row_key(
+ const Slice& encoded_key, bool with_seq_col, const RowsetIdUnorderedSet* rowset_ids,
+ RowLocation* row_location, uint32_t version,
+ std::unordered_map<RowsetId, SegmentCacheHandle, HashOfRowsetId>& segment_caches,
Review Comment:
yes，we need to refactor later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] liaoxin01 commented on pull request #20706: [Enhancement](merge-on-write) parallel calculate delete bitmap when tablet has multi segments
Posted by "liaoxin01 (via GitHub)" <gi...@apache.org>.
liaoxin01 commented on PR #20706:
URL: https://github.com/apache/doris/pull/20706#issuecomment-1590751896
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] liaoxin01 commented on a diff in pull request #20706: [Enhancement](merge-on-write) parallel calculate delete bitmap when tablet has multi segments
Posted by "liaoxin01 (via GitHub)" <gi...@apache.org>.
liaoxin01 commented on code in PR #20706:
URL: https://github.com/apache/doris/pull/20706#discussion_r1229222044
##########
be/src/service/point_query_executor.cpp:
##########
@@ -264,7 +265,7 @@ Status PointQueryExecutor::_lookup_row_key() {
// Get rowlocation and rowset, ctx._rowset_ptr will acquire wrap this ptr
auto rowset_ptr = std::make_unique<RowsetSharedPtr>();
st = (_tablet->lookup_row_key(_row_read_ctxs[i]._primary_key, true, nullptr, &location,
- INT32_MAX /*rethink?*/, rowset_ptr.get()));
+ INT32_MAX /*rethink?*/, segment_caches, rowset_ptr.get()));
Review Comment:
`_row_read_ctxs` may have multiple keys to lookup，so I think we also need a segment cache here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] liaoxin01 commented on pull request #20706: [Enhancement](merge-on-write) parallel calculate delete bitmap when tablet has multi segments
Posted by "liaoxin01 (via GitHub)" <gi...@apache.org>.
liaoxin01 commented on PR #20706:
URL: https://github.com/apache/doris/pull/20706#issuecomment-1588376285
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] liaoxin01 commented on pull request #20706: [Enhancement](merge-on-write) parallel calculate delete bitmap when tablet has multi segments
Posted by "liaoxin01 (via GitHub)" <gi...@apache.org>.
liaoxin01 commented on PR #20706:
URL: https://github.com/apache/doris/pull/20706#issuecomment-1588628871
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20706: [Enhancement](merge-on-write) parallel calculate delete bitmap when tablet has multi segments
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20706:
URL: https://github.com/apache/doris/pull/20706#issuecomment-1590754902
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org