You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "jixxiong (via GitHub)" <gi...@apache.org> on 2023/08/03 10:11:59 UTC
[GitHub] [doris] jixxiong opened a new pull request, #22566: [enhancement](MOW) refactor delete bitmap calculator and enhance the deletion bitmap calculation for complete updates
jixxiong opened a new pull request, #22566:
URL: https://github.com/apache/doris/pull/22566
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
TODO(xiongjixiang): fill this pr request
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #22566: [enhancement](MOW) refactor delete bitmap calculator and enhance the deletion bitmap calculation for full duplicate load
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #22566:
URL: https://github.com/apache/doris/pull/22566#issuecomment-1672865925
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.79 seconds
stream load tsv: 512 seconds loaded 74807831229 Bytes, about 139 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 30 seconds loaded 861443392 Bytes, about 27 MB/s
insert into select: 29.2 seconds inserted 10000000 Rows, about 342K ops/s
storage size: 17162059115 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] zhannngchen commented on a diff in pull request #22566: [enhancement](MOW) refactor delete bitmap calculator and enhance the deletion bitmap calculation for full duplicate load
Posted by "zhannngchen (via GitHub)" <gi...@apache.org>.
zhannngchen commented on code in PR #22566:
URL: https://github.com/apache/doris/pull/22566#discussion_r1289444110
##########
be/src/olap/delete_bitmap_calculator.cpp:
##########
@@ -83,116 +91,242 @@ Status MergeIndexDeleteBitmapCalculatorContext::_next_batch(size_t row_id) {
return Status::OK();
}
-bool MergeIndexDeleteBitmapCalculatorContext::Comparator::operator()(
- MergeIndexDeleteBitmapCalculatorContext* lhs,
- MergeIndexDeleteBitmapCalculatorContext* rhs) const {
+bool MergedPKIndexDeleteBitmapCalculatorContext::Comparator::operator()(
+ MergedPKIndexDeleteBitmapCalculatorContext* lhs,
+ MergedPKIndexDeleteBitmapCalculatorContext* rhs) const {
// std::proiroty_queue is a max heap, and function should return the result of `lhs < rhs`
// so if the result of the function is true, rhs will be popped before lhs
Slice key1, key2;
- RETURN_IF_ERROR(lhs->get_current_key(key1));
- RETURN_IF_ERROR(rhs->get_current_key(key2));
- if (_sequence_length == 0) {
- auto cmp_result = key1.compare(key2);
- // when key1 is the same as key2,
- // we want the one with greater segment id to be popped first
- return cmp_result ? (cmp_result > 0) : (lhs->segment_id() < rhs->segment_id());
- }
- // smaller key popped first
- auto key1_without_seq = Slice(key1.get_data(), key1.get_size() - _sequence_length);
- auto key2_without_seq = Slice(key2.get_data(), key2.get_size() - _sequence_length);
- auto cmp_result = key1_without_seq.compare(key2_without_seq);
- if (cmp_result != 0) {
- return cmp_result > 0;
- }
- // greater sequence value popped first
- auto key1_sequence_val =
- Slice(key1.get_data() + key1.get_size() - _sequence_length, _sequence_length);
- auto key2_sequence_val =
- Slice(key2.get_data() + key2.get_size() - _sequence_length, _sequence_length);
- cmp_result = key1_sequence_val.compare(key2_sequence_val);
- if (cmp_result != 0) {
- return cmp_result < 0;
- }
- // greater segment id popped first
+ lhs->get_current_key(&key1);
+ rhs->get_current_key(&key2);
+ // We start by comparing the key portions of the two records.
+ // If they are not equal, we want the record with the smaller key to be popped from the heap first.
+ // Therefore, we need to return the result of key1 > key2.
+ int key_cmp_result = compare_key(key1, key2);
+ if (LIKELY(key_cmp_result != 0)) {
+ return key_cmp_result > 0;
+ }
+ // If key portions of the two records are equal,
+ // we want the record with the larger sequence value to be popped from the heap first.
+ // Therefore, we need to return the result of seq1 < seq2.
+ int seq_cmp_result = compare_seq(key1, key2);
+ if (LIKELY(seq_cmp_result != 0)) {
+ return seq_cmp_result < 0;
+ }
+ // Otherwise, if they have the same key and sequence value,
+ // we want the record with the larger version value to be popped from the heap first.
+ // Therefore, we need to return the result of version1 < version2.
+ if (LIKELY(lhs->end_version() != rhs->end_version())) {
+ return lhs->end_version() < rhs->end_version();
+ }
+ // Lastly, if they have the same key, sequence value and version,
+ // we want the record with the larger segment id to be popped from the heap first.
+ // Therefore, we need to return the result of segid1 < segid2.
return lhs->segment_id() < rhs->segment_id();
}
-bool MergeIndexDeleteBitmapCalculatorContext::Comparator::is_key_same(Slice const& lhs,
- Slice const& rhs) const {
+int MergedPKIndexDeleteBitmapCalculatorContext::Comparator::compare_key(Slice const& lhs,
+ Slice const& rhs) const {
auto lhs_without_seq = Slice(lhs.get_data(), lhs.get_size() - _sequence_length);
auto rhs_without_seq = Slice(rhs.get_data(), rhs.get_size() - _sequence_length);
- return lhs_without_seq.compare(rhs_without_seq) == 0;
+ return lhs_without_seq.compare(rhs_without_seq);
+}
+
+int MergedPKIndexDeleteBitmapCalculatorContext::Comparator::compare_seq(Slice const& lhs,
+ Slice const& rhs) const {
+ if (LIKELY(_sequence_length == 0)) {
+ return 0;
+ }
+ auto lhs_seq = Slice(lhs.get_data() + lhs.get_size() - _sequence_length, _sequence_length);
+ auto rhs_seq = Slice(rhs.get_data() + rhs.get_size() - _sequence_length, _sequence_length);
+ return lhs_seq.compare(rhs_seq);
+}
+
+bool MergedPKIndexDeleteBitmapCalculatorContext::Comparator::is_key_same(Slice const& lhs,
+ Slice const& rhs) const {
+ return compare_key(lhs, rhs) == 0;
}
-Status MergeIndexDeleteBitmapCalculator::init(RowsetId rowset_id,
- std::vector<SegmentSharedPtr> const& segments,
- size_t seq_col_length, size_t max_batch_size) {
- _rowset_id = rowset_id;
+Status MergedPKIndexDeleteBitmapCalculator::init(std::vector<SegmentSharedPtr> const& segments,
+ const std::vector<int64_t>* end_versions,
+ size_t num_delta_segments, size_t seq_col_length,
+ size_t max_batch_size) {
+ // The first `num_delta_segments` segments are the delta data, others are the base data.
+ DCHECK_LE(num_delta_segments, segments.size());
_seq_col_length = seq_col_length;
- _comparator = MergeIndexDeleteBitmapCalculatorContext::Comparator(seq_col_length);
- _contexts.reserve(segments.size());
+ _comparator = MergedPKIndexDeleteBitmapCalculatorContext::Comparator(seq_col_length);
+ size_t num_segments = segments.size();
+ _contexts.reserve(num_segments);
_heap = std::make_unique<Heap>(_comparator);
- for (auto& segment : segments) {
+ for (size_t i = 0; i < num_segments; ++i) {
+ auto&& segment = segments[i];
+ int64_t end_version = end_versions ? (*end_versions)[i] : 0;
RETURN_IF_ERROR(segment->load_index());
auto pk_idx = segment->get_primary_key_index();
std::unique_ptr<segment_v2::IndexedColumnIterator> index;
RETURN_IF_ERROR(pk_idx->new_iterator(&index));
auto index_type = vectorized::DataTypeFactory::instance().create_data_type(
pk_idx->type_info()->type(), 1, 0);
- _contexts.emplace_back(std::move(index), index_type, segment->id(), pk_idx->num_rows());
+ bool is_delta_segment = i < num_delta_segments;
+ _contexts.emplace_back(std::move(index), index_type, segment->rowset_id(), end_version,
+ segment->id(), is_delta_segment, pk_idx->num_rows(), max_batch_size);
Review Comment:
in the ctor, the 6th parameter is `is_base_segment` rather than `is_delta_segment`, it's not match
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] zhannngchen commented on a diff in pull request #22566: [enhancement](MOW) refactor delete bitmap calculator and enhance the deletion bitmap calculation for full duplicate load
Posted by "zhannngchen (via GitHub)" <gi...@apache.org>.
zhannngchen commented on code in PR #22566:
URL: https://github.com/apache/doris/pull/22566#discussion_r1288184138
##########
be/src/olap/delete_bitmap_calculator.cpp:
##########
@@ -83,116 +91,209 @@ Status MergeIndexDeleteBitmapCalculatorContext::_next_batch(size_t row_id) {
return Status::OK();
}
-bool MergeIndexDeleteBitmapCalculatorContext::Comparator::operator()(
- MergeIndexDeleteBitmapCalculatorContext* lhs,
- MergeIndexDeleteBitmapCalculatorContext* rhs) const {
+bool MergedPKIndexDeleteBitmapCalculatorContext::Comparator::operator()(
+ MergedPKIndexDeleteBitmapCalculatorContext* lhs,
+ MergedPKIndexDeleteBitmapCalculatorContext* rhs) const {
// std::proiroty_queue is a max heap, and function should return the result of `lhs < rhs`
// so if the result of the function is true, rhs will be popped before lhs
Slice key1, key2;
- RETURN_IF_ERROR(lhs->get_current_key(key1));
- RETURN_IF_ERROR(rhs->get_current_key(key2));
- if (_sequence_length == 0) {
- auto cmp_result = key1.compare(key2);
- // when key1 is the same as key2,
- // we want the one with greater segment id to be popped first
- return cmp_result ? (cmp_result > 0) : (lhs->segment_id() < rhs->segment_id());
- }
- // smaller key popped first
- auto key1_without_seq = Slice(key1.get_data(), key1.get_size() - _sequence_length);
- auto key2_without_seq = Slice(key2.get_data(), key2.get_size() - _sequence_length);
- auto cmp_result = key1_without_seq.compare(key2_without_seq);
- if (cmp_result != 0) {
- return cmp_result > 0;
- }
- // greater sequence value popped first
- auto key1_sequence_val =
- Slice(key1.get_data() + key1.get_size() - _sequence_length, _sequence_length);
- auto key2_sequence_val =
- Slice(key2.get_data() + key2.get_size() - _sequence_length, _sequence_length);
- cmp_result = key1_sequence_val.compare(key2_sequence_val);
- if (cmp_result != 0) {
- return cmp_result < 0;
- }
- // greater segment id popped first
+ Status st;
+ st = lhs->get_current_key(&key1);
+ CHECK(LIKELY(st.ok())) << fmt::format("Reading key1 but get st = {}", st);
+ st = rhs->get_current_key(&key2);
+ CHECK(LIKELY(st.ok())) << fmt::format("Reading key2 but get st = {}", st);
+ int key_cmp_result = compare_key(key1, key2);
+ if (LIKELY(key_cmp_result != 0)) {
+ return key_cmp_result > 0;
+ }
+ int seq_cmp_result = compare_seq(key1, key2);
+ if (LIKELY(seq_cmp_result != 0)) {
+ return seq_cmp_result < 0;
+ }
+ if (LIKELY(lhs->end_version() != rhs->end_version())) {
+ return lhs->end_version() < rhs->end_version();
+ }
return lhs->segment_id() < rhs->segment_id();
}
-bool MergeIndexDeleteBitmapCalculatorContext::Comparator::is_key_same(Slice const& lhs,
- Slice const& rhs) const {
+int MergedPKIndexDeleteBitmapCalculatorContext::Comparator::compare_key(Slice const& lhs,
+ Slice const& rhs) const {
auto lhs_without_seq = Slice(lhs.get_data(), lhs.get_size() - _sequence_length);
auto rhs_without_seq = Slice(rhs.get_data(), rhs.get_size() - _sequence_length);
- return lhs_without_seq.compare(rhs_without_seq) == 0;
+ return lhs_without_seq.compare(rhs_without_seq);
+}
+
+int MergedPKIndexDeleteBitmapCalculatorContext::Comparator::compare_seq(Slice const& lhs,
+ Slice const& rhs) const {
+ if (UNLIKELY(_sequence_length == 0)) {
Review Comment:
Should be `LIKELY`, most user don't use sequence column
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #22566: [enhancement](MOW) refactor delete bitmap calculator and enhance the deletion bitmap calculation for full duplicate load
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #22566:
URL: https://github.com/apache/doris/pull/22566#issuecomment-1674152775
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.32 seconds
stream load tsv: 517 seconds loaded 74807831229 Bytes, about 137 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.4 seconds inserted 10000000 Rows, about 340K ops/s
storage size: 17162182728 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] zhannngchen commented on a diff in pull request #22566: [enhancement](MOW) refactor delete bitmap calculator and enhance the deletion bitmap calculation for full duplicate load
Posted by "zhannngchen (via GitHub)" <gi...@apache.org>.
zhannngchen commented on code in PR #22566:
URL: https://github.com/apache/doris/pull/22566#discussion_r1288028685
##########
be/src/olap/delete_bitmap_calculator.cpp:
##########
@@ -83,116 +91,209 @@ Status MergeIndexDeleteBitmapCalculatorContext::_next_batch(size_t row_id) {
return Status::OK();
}
-bool MergeIndexDeleteBitmapCalculatorContext::Comparator::operator()(
- MergeIndexDeleteBitmapCalculatorContext* lhs,
- MergeIndexDeleteBitmapCalculatorContext* rhs) const {
+bool MergedPKIndexDeleteBitmapCalculatorContext::Comparator::operator()(
+ MergedPKIndexDeleteBitmapCalculatorContext* lhs,
+ MergedPKIndexDeleteBitmapCalculatorContext* rhs) const {
// std::proiroty_queue is a max heap, and function should return the result of `lhs < rhs`
// so if the result of the function is true, rhs will be popped before lhs
Slice key1, key2;
- RETURN_IF_ERROR(lhs->get_current_key(key1));
- RETURN_IF_ERROR(rhs->get_current_key(key2));
- if (_sequence_length == 0) {
- auto cmp_result = key1.compare(key2);
- // when key1 is the same as key2,
- // we want the one with greater segment id to be popped first
- return cmp_result ? (cmp_result > 0) : (lhs->segment_id() < rhs->segment_id());
- }
- // smaller key popped first
- auto key1_without_seq = Slice(key1.get_data(), key1.get_size() - _sequence_length);
- auto key2_without_seq = Slice(key2.get_data(), key2.get_size() - _sequence_length);
- auto cmp_result = key1_without_seq.compare(key2_without_seq);
- if (cmp_result != 0) {
- return cmp_result > 0;
- }
- // greater sequence value popped first
- auto key1_sequence_val =
- Slice(key1.get_data() + key1.get_size() - _sequence_length, _sequence_length);
- auto key2_sequence_val =
- Slice(key2.get_data() + key2.get_size() - _sequence_length, _sequence_length);
- cmp_result = key1_sequence_val.compare(key2_sequence_val);
- if (cmp_result != 0) {
- return cmp_result < 0;
- }
- // greater segment id popped first
+ Status st;
+ st = lhs->get_current_key(&key1);
+ CHECK(LIKELY(st.ok())) << fmt::format("Reading key1 but get st = {}", st);
+ st = rhs->get_current_key(&key2);
+ CHECK(LIKELY(st.ok())) << fmt::format("Reading key2 but get st = {}", st);
+ int key_cmp_result = compare_key(key1, key2);
+ if (LIKELY(key_cmp_result != 0)) {
+ return key_cmp_result > 0;
+ }
+ int seq_cmp_result = compare_seq(key1, key2);
+ if (LIKELY(seq_cmp_result != 0)) {
+ return seq_cmp_result < 0;
+ }
+ if (LIKELY(lhs->end_version() != rhs->end_version())) {
+ return lhs->end_version() < rhs->end_version();
+ }
return lhs->segment_id() < rhs->segment_id();
}
-bool MergeIndexDeleteBitmapCalculatorContext::Comparator::is_key_same(Slice const& lhs,
- Slice const& rhs) const {
+int MergedPKIndexDeleteBitmapCalculatorContext::Comparator::compare_key(Slice const& lhs,
+ Slice const& rhs) const {
auto lhs_without_seq = Slice(lhs.get_data(), lhs.get_size() - _sequence_length);
auto rhs_without_seq = Slice(rhs.get_data(), rhs.get_size() - _sequence_length);
- return lhs_without_seq.compare(rhs_without_seq) == 0;
+ return lhs_without_seq.compare(rhs_without_seq);
+}
+
+int MergedPKIndexDeleteBitmapCalculatorContext::Comparator::compare_seq(Slice const& lhs,
+ Slice const& rhs) const {
+ if (UNLIKELY(_sequence_length == 0)) {
+ return 0;
+ }
+ auto lhs_seq = Slice(lhs.get_data() + lhs.get_size() - _sequence_length, _sequence_length);
+ auto rhs_seq = Slice(rhs.get_data() + rhs.get_size() - _sequence_length, _sequence_length);
+ return lhs_seq.compare(rhs_seq);
+}
+
+bool MergedPKIndexDeleteBitmapCalculatorContext::Comparator::is_key_same(Slice const& lhs,
+ Slice const& rhs) const {
+ return compare_key(lhs, rhs) == 0;
}
-Status MergeIndexDeleteBitmapCalculator::init(RowsetId rowset_id,
- std::vector<SegmentSharedPtr> const& segments,
- size_t seq_col_length, size_t max_batch_size) {
- _rowset_id = rowset_id;
+Status MergedPKIndexDeleteBitmapCalculator::init(std::vector<SegmentSharedPtr> const& segments,
+ const std::vector<int64_t>* end_versions,
+ size_t num_base_segments, size_t seq_col_length,
+ size_t max_batch_size) {
_seq_col_length = seq_col_length;
- _comparator = MergeIndexDeleteBitmapCalculatorContext::Comparator(seq_col_length);
- _contexts.reserve(segments.size());
+ _comparator = MergedPKIndexDeleteBitmapCalculatorContext::Comparator(seq_col_length);
+ size_t num_segments = segments.size();
+ _contexts.reserve(num_segments);
_heap = std::make_unique<Heap>(_comparator);
- for (auto& segment : segments) {
+ for (size_t i = 0; i < num_segments; ++i) {
+ auto&& segment = segments[i];
+ int64_t end_version = end_versions ? (*end_versions)[i] : 0;
+ segment->rowset_id();
Review Comment:
useless?
##########
be/src/olap/delete_bitmap_calculator.cpp:
##########
@@ -83,116 +91,209 @@ Status MergeIndexDeleteBitmapCalculatorContext::_next_batch(size_t row_id) {
return Status::OK();
}
-bool MergeIndexDeleteBitmapCalculatorContext::Comparator::operator()(
- MergeIndexDeleteBitmapCalculatorContext* lhs,
- MergeIndexDeleteBitmapCalculatorContext* rhs) const {
+bool MergedPKIndexDeleteBitmapCalculatorContext::Comparator::operator()(
+ MergedPKIndexDeleteBitmapCalculatorContext* lhs,
+ MergedPKIndexDeleteBitmapCalculatorContext* rhs) const {
// std::proiroty_queue is a max heap, and function should return the result of `lhs < rhs`
// so if the result of the function is true, rhs will be popped before lhs
Slice key1, key2;
- RETURN_IF_ERROR(lhs->get_current_key(key1));
- RETURN_IF_ERROR(rhs->get_current_key(key2));
- if (_sequence_length == 0) {
- auto cmp_result = key1.compare(key2);
- // when key1 is the same as key2,
- // we want the one with greater segment id to be popped first
- return cmp_result ? (cmp_result > 0) : (lhs->segment_id() < rhs->segment_id());
- }
- // smaller key popped first
- auto key1_without_seq = Slice(key1.get_data(), key1.get_size() - _sequence_length);
- auto key2_without_seq = Slice(key2.get_data(), key2.get_size() - _sequence_length);
- auto cmp_result = key1_without_seq.compare(key2_without_seq);
- if (cmp_result != 0) {
- return cmp_result > 0;
- }
- // greater sequence value popped first
- auto key1_sequence_val =
- Slice(key1.get_data() + key1.get_size() - _sequence_length, _sequence_length);
- auto key2_sequence_val =
- Slice(key2.get_data() + key2.get_size() - _sequence_length, _sequence_length);
- cmp_result = key1_sequence_val.compare(key2_sequence_val);
- if (cmp_result != 0) {
- return cmp_result < 0;
- }
- // greater segment id popped first
+ Status st;
+ st = lhs->get_current_key(&key1);
+ CHECK(LIKELY(st.ok())) << fmt::format("Reading key1 but get st = {}", st);
+ st = rhs->get_current_key(&key2);
+ CHECK(LIKELY(st.ok())) << fmt::format("Reading key2 but get st = {}", st);
+ int key_cmp_result = compare_key(key1, key2);
+ if (LIKELY(key_cmp_result != 0)) {
+ return key_cmp_result > 0;
+ }
+ int seq_cmp_result = compare_seq(key1, key2);
+ if (LIKELY(seq_cmp_result != 0)) {
+ return seq_cmp_result < 0;
+ }
+ if (LIKELY(lhs->end_version() != rhs->end_version())) {
+ return lhs->end_version() < rhs->end_version();
+ }
return lhs->segment_id() < rhs->segment_id();
}
-bool MergeIndexDeleteBitmapCalculatorContext::Comparator::is_key_same(Slice const& lhs,
- Slice const& rhs) const {
+int MergedPKIndexDeleteBitmapCalculatorContext::Comparator::compare_key(Slice const& lhs,
+ Slice const& rhs) const {
auto lhs_without_seq = Slice(lhs.get_data(), lhs.get_size() - _sequence_length);
auto rhs_without_seq = Slice(rhs.get_data(), rhs.get_size() - _sequence_length);
- return lhs_without_seq.compare(rhs_without_seq) == 0;
+ return lhs_without_seq.compare(rhs_without_seq);
+}
+
+int MergedPKIndexDeleteBitmapCalculatorContext::Comparator::compare_seq(Slice const& lhs,
+ Slice const& rhs) const {
+ if (UNLIKELY(_sequence_length == 0)) {
+ return 0;
+ }
+ auto lhs_seq = Slice(lhs.get_data() + lhs.get_size() - _sequence_length, _sequence_length);
+ auto rhs_seq = Slice(rhs.get_data() + rhs.get_size() - _sequence_length, _sequence_length);
+ return lhs_seq.compare(rhs_seq);
+}
+
+bool MergedPKIndexDeleteBitmapCalculatorContext::Comparator::is_key_same(Slice const& lhs,
+ Slice const& rhs) const {
+ return compare_key(lhs, rhs) == 0;
}
-Status MergeIndexDeleteBitmapCalculator::init(RowsetId rowset_id,
- std::vector<SegmentSharedPtr> const& segments,
- size_t seq_col_length, size_t max_batch_size) {
- _rowset_id = rowset_id;
+Status MergedPKIndexDeleteBitmapCalculator::init(std::vector<SegmentSharedPtr> const& segments,
+ const std::vector<int64_t>* end_versions,
+ size_t num_base_segments, size_t seq_col_length,
+ size_t max_batch_size) {
_seq_col_length = seq_col_length;
- _comparator = MergeIndexDeleteBitmapCalculatorContext::Comparator(seq_col_length);
- _contexts.reserve(segments.size());
+ _comparator = MergedPKIndexDeleteBitmapCalculatorContext::Comparator(seq_col_length);
+ size_t num_segments = segments.size();
+ _contexts.reserve(num_segments);
_heap = std::make_unique<Heap>(_comparator);
- for (auto& segment : segments) {
+ for (size_t i = 0; i < num_segments; ++i) {
+ auto&& segment = segments[i];
+ int64_t end_version = end_versions ? (*end_versions)[i] : 0;
+ segment->rowset_id();
RETURN_IF_ERROR(segment->load_index());
auto pk_idx = segment->get_primary_key_index();
std::unique_ptr<segment_v2::IndexedColumnIterator> index;
RETURN_IF_ERROR(pk_idx->new_iterator(&index));
auto index_type = vectorized::DataTypeFactory::instance().create_data_type(
pk_idx->type_info()->type(), 1, 0);
- _contexts.emplace_back(std::move(index), index_type, segment->id(), pk_idx->num_rows());
+ bool is_base_segment = i < num_base_segments;
+ _contexts.emplace_back(std::move(index), index_type, segment->rowset_id(), end_version,
+ segment->id(), is_base_segment, pk_idx->num_rows(), max_batch_size);
_heap->push(&_contexts.back());
}
return Status::OK();
}
-Status MergeIndexDeleteBitmapCalculator::calculate_one(RowLocation& loc) {
- // get the location of a out-of-date row
- while (!_heap->empty()) {
- auto cur_ctx = _heap->top();
+Status MergedPKIndexDeleteBitmapCalculator::init(
+ std::vector<SegmentSharedPtr> const& segments,
+ const std::vector<RowsetSharedPtr>* specified_rowsets,
+ bool calc_delete_bitmap_between_segments, size_t seq_col_length, size_t max_batch_size) {
Review Comment:
parameter `calc_delete_bitmap_between_segments` seems not necessary, you can use `specified_rowsets == nullptr` instead?
##########
be/src/olap/delete_bitmap_calculator.cpp:
##########
@@ -83,116 +91,209 @@ Status MergeIndexDeleteBitmapCalculatorContext::_next_batch(size_t row_id) {
return Status::OK();
}
-bool MergeIndexDeleteBitmapCalculatorContext::Comparator::operator()(
- MergeIndexDeleteBitmapCalculatorContext* lhs,
- MergeIndexDeleteBitmapCalculatorContext* rhs) const {
+bool MergedPKIndexDeleteBitmapCalculatorContext::Comparator::operator()(
+ MergedPKIndexDeleteBitmapCalculatorContext* lhs,
+ MergedPKIndexDeleteBitmapCalculatorContext* rhs) const {
// std::proiroty_queue is a max heap, and function should return the result of `lhs < rhs`
// so if the result of the function is true, rhs will be popped before lhs
Slice key1, key2;
- RETURN_IF_ERROR(lhs->get_current_key(key1));
- RETURN_IF_ERROR(rhs->get_current_key(key2));
- if (_sequence_length == 0) {
- auto cmp_result = key1.compare(key2);
- // when key1 is the same as key2,
- // we want the one with greater segment id to be popped first
- return cmp_result ? (cmp_result > 0) : (lhs->segment_id() < rhs->segment_id());
- }
- // smaller key popped first
- auto key1_without_seq = Slice(key1.get_data(), key1.get_size() - _sequence_length);
- auto key2_without_seq = Slice(key2.get_data(), key2.get_size() - _sequence_length);
- auto cmp_result = key1_without_seq.compare(key2_without_seq);
- if (cmp_result != 0) {
- return cmp_result > 0;
- }
- // greater sequence value popped first
- auto key1_sequence_val =
- Slice(key1.get_data() + key1.get_size() - _sequence_length, _sequence_length);
- auto key2_sequence_val =
- Slice(key2.get_data() + key2.get_size() - _sequence_length, _sequence_length);
- cmp_result = key1_sequence_val.compare(key2_sequence_val);
- if (cmp_result != 0) {
- return cmp_result < 0;
- }
- // greater segment id popped first
+ Status st;
+ st = lhs->get_current_key(&key1);
+ CHECK(LIKELY(st.ok())) << fmt::format("Reading key1 but get st = {}", st);
+ st = rhs->get_current_key(&key2);
+ CHECK(LIKELY(st.ok())) << fmt::format("Reading key2 but get st = {}", st);
+ int key_cmp_result = compare_key(key1, key2);
+ if (LIKELY(key_cmp_result != 0)) {
+ return key_cmp_result > 0;
+ }
+ int seq_cmp_result = compare_seq(key1, key2);
+ if (LIKELY(seq_cmp_result != 0)) {
+ return seq_cmp_result < 0;
+ }
+ if (LIKELY(lhs->end_version() != rhs->end_version())) {
+ return lhs->end_version() < rhs->end_version();
+ }
return lhs->segment_id() < rhs->segment_id();
}
-bool MergeIndexDeleteBitmapCalculatorContext::Comparator::is_key_same(Slice const& lhs,
- Slice const& rhs) const {
+int MergedPKIndexDeleteBitmapCalculatorContext::Comparator::compare_key(Slice const& lhs,
+ Slice const& rhs) const {
auto lhs_without_seq = Slice(lhs.get_data(), lhs.get_size() - _sequence_length);
auto rhs_without_seq = Slice(rhs.get_data(), rhs.get_size() - _sequence_length);
- return lhs_without_seq.compare(rhs_without_seq) == 0;
+ return lhs_without_seq.compare(rhs_without_seq);
+}
+
+int MergedPKIndexDeleteBitmapCalculatorContext::Comparator::compare_seq(Slice const& lhs,
+ Slice const& rhs) const {
+ if (UNLIKELY(_sequence_length == 0)) {
+ return 0;
+ }
+ auto lhs_seq = Slice(lhs.get_data() + lhs.get_size() - _sequence_length, _sequence_length);
+ auto rhs_seq = Slice(rhs.get_data() + rhs.get_size() - _sequence_length, _sequence_length);
+ return lhs_seq.compare(rhs_seq);
+}
+
+bool MergedPKIndexDeleteBitmapCalculatorContext::Comparator::is_key_same(Slice const& lhs,
+ Slice const& rhs) const {
+ return compare_key(lhs, rhs) == 0;
}
-Status MergeIndexDeleteBitmapCalculator::init(RowsetId rowset_id,
- std::vector<SegmentSharedPtr> const& segments,
- size_t seq_col_length, size_t max_batch_size) {
- _rowset_id = rowset_id;
+Status MergedPKIndexDeleteBitmapCalculator::init(std::vector<SegmentSharedPtr> const& segments,
+ const std::vector<int64_t>* end_versions,
+ size_t num_base_segments, size_t seq_col_length,
+ size_t max_batch_size) {
_seq_col_length = seq_col_length;
- _comparator = MergeIndexDeleteBitmapCalculatorContext::Comparator(seq_col_length);
- _contexts.reserve(segments.size());
+ _comparator = MergedPKIndexDeleteBitmapCalculatorContext::Comparator(seq_col_length);
+ size_t num_segments = segments.size();
Review Comment:
Add a DCHECK_LT(num_base_segments, num_segments);
##########
be/src/olap/delete_bitmap_calculator.cpp:
##########
@@ -83,116 +91,209 @@ Status MergeIndexDeleteBitmapCalculatorContext::_next_batch(size_t row_id) {
return Status::OK();
}
-bool MergeIndexDeleteBitmapCalculatorContext::Comparator::operator()(
- MergeIndexDeleteBitmapCalculatorContext* lhs,
- MergeIndexDeleteBitmapCalculatorContext* rhs) const {
+bool MergedPKIndexDeleteBitmapCalculatorContext::Comparator::operator()(
+ MergedPKIndexDeleteBitmapCalculatorContext* lhs,
+ MergedPKIndexDeleteBitmapCalculatorContext* rhs) const {
// std::proiroty_queue is a max heap, and function should return the result of `lhs < rhs`
// so if the result of the function is true, rhs will be popped before lhs
Slice key1, key2;
- RETURN_IF_ERROR(lhs->get_current_key(key1));
- RETURN_IF_ERROR(rhs->get_current_key(key2));
- if (_sequence_length == 0) {
- auto cmp_result = key1.compare(key2);
- // when key1 is the same as key2,
- // we want the one with greater segment id to be popped first
- return cmp_result ? (cmp_result > 0) : (lhs->segment_id() < rhs->segment_id());
- }
- // smaller key popped first
- auto key1_without_seq = Slice(key1.get_data(), key1.get_size() - _sequence_length);
- auto key2_without_seq = Slice(key2.get_data(), key2.get_size() - _sequence_length);
- auto cmp_result = key1_without_seq.compare(key2_without_seq);
- if (cmp_result != 0) {
- return cmp_result > 0;
- }
- // greater sequence value popped first
- auto key1_sequence_val =
- Slice(key1.get_data() + key1.get_size() - _sequence_length, _sequence_length);
- auto key2_sequence_val =
- Slice(key2.get_data() + key2.get_size() - _sequence_length, _sequence_length);
- cmp_result = key1_sequence_val.compare(key2_sequence_val);
- if (cmp_result != 0) {
- return cmp_result < 0;
- }
- // greater segment id popped first
+ Status st;
+ st = lhs->get_current_key(&key1);
+ CHECK(LIKELY(st.ok())) << fmt::format("Reading key1 but get st = {}", st);
Review Comment:
use DCHECK
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22566: [enhancement](MOW) refactor delete bitmap calculator and enhance the deletion bitmap calculation for full duplicate load
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #22566:
URL: https://github.com/apache/doris/pull/22566#issuecomment-1671063220
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22566: [enhancement](MOW) refactor delete bitmap calculator and enhance the deletion bitmap calculation for full duplicate load
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #22566:
URL: https://github.com/apache/doris/pull/22566#issuecomment-1671065460
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22566: [enhancement](MOW) refactor delete bitmap calculator and enhance the deletion bitmap calculation for full duplicate load
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #22566:
URL: https://github.com/apache/doris/pull/22566#issuecomment-1664933726
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] jixxiong commented on pull request #22566: [enhancement](MOW) refactor delete bitmap calculator and enhance the deletion bitmap calculation for full duplicate load
Posted by "jixxiong (via GitHub)" <gi...@apache.org>.
jixxiong commented on PR #22566:
URL: https://github.com/apache/doris/pull/22566#issuecomment-1672810672
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] jixxiong commented on pull request #22566: [enhancement](MOW) refactor delete bitmap calculator and enhance the deletion bitmap calculation for full duplicate load
Posted by "jixxiong (via GitHub)" <gi...@apache.org>.
jixxiong commented on PR #22566:
URL: https://github.com/apache/doris/pull/22566#issuecomment-1674107784
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22566: [enhancement](MOW) refactor delete bitmap calculator and enhance the deletion bitmap calculation for full duplicate load
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #22566:
URL: https://github.com/apache/doris/pull/22566#issuecomment-1671064742
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22566: [enhancement](MOW) refactor delete bitmap calculator and enhance the deletion bitmap calculation for full duplicate load
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #22566:
URL: https://github.com/apache/doris/pull/22566#issuecomment-1671046601
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] jixxiong commented on pull request #22566: [enhancement](MOW) refactor delete bitmap calculator and enhance the deletion bitmap calculation for full duplicate load
Posted by "jixxiong (via GitHub)" <gi...@apache.org>.
jixxiong commented on PR #22566:
URL: https://github.com/apache/doris/pull/22566#issuecomment-1671068416
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22566: [enhancement](MOW) refactor delete bitmap calculator and enhance the deletion bitmap calculation for full duplicate load
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #22566:
URL: https://github.com/apache/doris/pull/22566#issuecomment-1664933822
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] jixxiong commented on pull request #22566: [enhancement](MOW) refactor delete bitmap calculator and enhance the deletion bitmap calculation for full duplicate load
Posted by "jixxiong (via GitHub)" <gi...@apache.org>.
jixxiong commented on PR #22566:
URL: https://github.com/apache/doris/pull/22566#issuecomment-1676508717
run p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](MOW) refactor delete bitmap calculator and enhance the deletion bitmap calculation for full duplicate load [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #22566:
URL: https://github.com/apache/doris/pull/22566#issuecomment-1937363438
We're closing this PR because it hasn't been updated in a while.
This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and feel free a maintainer to remove the Stale tag!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] jixxiong commented on pull request #22566: [enhancement](MOW) refactor delete bitmap calculator and enhance the deletion bitmap calculation for full duplicate load
Posted by "jixxiong (via GitHub)" <gi...@apache.org>.
jixxiong commented on PR #22566:
URL: https://github.com/apache/doris/pull/22566#issuecomment-1671207429
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22566: [enhancement](MOW) refactor delete bitmap calculator and enhance the deletion bitmap calculation for full duplicate load
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #22566:
URL: https://github.com/apache/doris/pull/22566#issuecomment-1670589907
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22566: [enhancement](MOW) refactor delete bitmap calculator and enhance the deletion bitmap calculation for full duplicate load
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #22566:
URL: https://github.com/apache/doris/pull/22566#issuecomment-1677138242
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22566: [enhancement](MOW) refactor delete bitmap calculator and enhance the deletion bitmap calculation for complete updates
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #22566:
URL: https://github.com/apache/doris/pull/22566#issuecomment-1663724914
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] zhannngchen commented on a diff in pull request #22566: [enhancement](MOW) refactor delete bitmap calculator and enhance the deletion bitmap calculation for full duplicate load
Posted by "zhannngchen (via GitHub)" <gi...@apache.org>.
zhannngchen commented on code in PR #22566:
URL: https://github.com/apache/doris/pull/22566#discussion_r1288187978
##########
be/src/olap/delete_bitmap_calculator.h:
##########
@@ -82,31 +97,41 @@ class MergeIndexDeleteBitmapCalculatorContext {
size_t _cur_row_id {0};
size_t const _num_rows;
size_t const _max_batch_size;
+ size_t const _end_version;
+ bool const _base_segment;
+ RowsetId const _rowset_id;
int32_t const _segment_id;
- bool _excat_match {false};
+ bool _excat_match;
};
-class MergeIndexDeleteBitmapCalculator {
+class MergedPKIndexDeleteBitmapCalculator {
public:
- MergeIndexDeleteBitmapCalculator() = default;
-
- Status init(RowsetId rowset_id, std::vector<SegmentSharedPtr> const& segments,
- size_t seq_col_length = 0, size_t max_batch_size = 1024);
+ MergedPKIndexDeleteBitmapCalculator() = default;
- Status calculate_one(RowLocation& loc);
+ Status init(std::vector<SegmentSharedPtr> const& segments,
Review Comment:
There's 2 ways of using `MergedPKIndexDeleteBitmapCalculator`, you need to explain how to use it for these 2 ways in comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22566: [enhancement](MOW) refactor delete bitmap calculator and enhance the deletion bitmap calculation for full duplicate load
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #22566:
URL: https://github.com/apache/doris/pull/22566#issuecomment-1674110762
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] jixxiong commented on pull request #22566: [enhancement](MOW) refactor delete bitmap calculator and enhance the deletion bitmap calculation for full duplicate load
Posted by "jixxiong (via GitHub)" <gi...@apache.org>.
jixxiong commented on PR #22566:
URL: https://github.com/apache/doris/pull/22566#issuecomment-1670591081
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22566: [enhancement](MOW) refactor delete bitmap calculator and enhance the deletion bitmap calculation for full duplicate load
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #22566:
URL: https://github.com/apache/doris/pull/22566#issuecomment-1671024504
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] jixxiong commented on pull request #22566: [enhancement](MOW) refactor delete bitmap calculator and enhance the deletion bitmap calculation for full duplicate load
Posted by "jixxiong (via GitHub)" <gi...@apache.org>.
jixxiong commented on PR #22566:
URL: https://github.com/apache/doris/pull/22566#issuecomment-1676509189
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22566: [enhancement](MOW) refactor delete bitmap calculator and enhance the deletion bitmap calculation for full duplicate load
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #22566:
URL: https://github.com/apache/doris/pull/22566#issuecomment-1671047482
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22566: [enhancement](MOW) refactor delete bitmap calculator and enhance the deletion bitmap calculation for full duplicate load
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #22566:
URL: https://github.com/apache/doris/pull/22566#issuecomment-1672792169
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] jixxiong commented on pull request #22566: [enhancement](MOW) refactor delete bitmap calculator and enhance the deletion bitmap calculation for full duplicate load
Posted by "jixxiong (via GitHub)" <gi...@apache.org>.
jixxiong commented on PR #22566:
URL: https://github.com/apache/doris/pull/22566#issuecomment-1677138950
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22566: [enhancement](MOW) refactor delete bitmap calculator and enhance the deletion bitmap calculation for full duplicate load
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #22566:
URL: https://github.com/apache/doris/pull/22566#issuecomment-1671029431
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #22566: [enhancement](MOW) refactor delete bitmap calculator and enhance the deletion bitmap calculation for full duplicate load
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #22566:
URL: https://github.com/apache/doris/pull/22566#issuecomment-1671121967
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.46 seconds
stream load tsv: 510 seconds loaded 74807831229 Bytes, about 139 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 30 seconds loaded 861443392 Bytes, about 27 MB/s
insert into select: 29.0 seconds inserted 10000000 Rows, about 344K ops/s
storage size: 17162165275 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22566: [enhancement](MOW) refactor delete bitmap calculator and enhance the deletion bitmap calculation for full duplicate load
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #22566:
URL: https://github.com/apache/doris/pull/22566#issuecomment-1672803648
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #22566: [enhancement](MOW) refactor delete bitmap calculator and enhance the deletion bitmap calculation for full duplicate load
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #22566:
URL: https://github.com/apache/doris/pull/22566#issuecomment-1665451465
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 48.37 seconds
stream load tsv: 509 seconds loaded 74807831229 Bytes, about 140 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.1 seconds inserted 10000000 Rows, about 343K ops/s
storage size: 17162308626 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] jixxiong commented on pull request #22566: [enhancement](MOW) refactor delete bitmap calculator and enhance the deletion bitmap calculation for full duplicate load
Posted by "jixxiong (via GitHub)" <gi...@apache.org>.
jixxiong commented on PR #22566:
URL: https://github.com/apache/doris/pull/22566#issuecomment-1665357070
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #22566: [enhancement](MOW) refactor delete bitmap calculator and enhance the deletion bitmap calculation for full duplicate load
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #22566:
URL: https://github.com/apache/doris/pull/22566#issuecomment-1676550554
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.83 seconds
stream load tsv: 517 seconds loaded 74807831229 Bytes, about 137 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.1 seconds inserted 10000000 Rows, about 343K ops/s
storage size: 17162304040 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #22566: [enhancement](MOW) refactor delete bitmap calculator and enhance the deletion bitmap calculation for full duplicate load
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #22566:
URL: https://github.com/apache/doris/pull/22566#issuecomment-1670633113
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.22 seconds
stream load tsv: 514 seconds loaded 74807831229 Bytes, about 138 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.1 seconds inserted 10000000 Rows, about 343K ops/s
storage size: 17162229976 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22566: [enhancement](MOW) refactor delete bitmap calculator and enhance the deletion bitmap calculation for full duplicate load
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #22566:
URL: https://github.com/apache/doris/pull/22566#issuecomment-1671207110
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #22566: [enhancement](MOW) refactor delete bitmap calculator and enhance the deletion bitmap calculation for full duplicate load
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #22566:
URL: https://github.com/apache/doris/pull/22566#issuecomment-1671256073
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.66 seconds
stream load tsv: 510 seconds loaded 74807831229 Bytes, about 139 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.3 seconds inserted 10000000 Rows, about 341K ops/s
storage size: 17162109814 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22566: [enhancement](MOW) refactor delete bitmap calculator and enhance the deletion bitmap calculation for full duplicate load
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #22566:
URL: https://github.com/apache/doris/pull/22566#issuecomment-1665326087
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](MOW) refactor delete bitmap calculator and enhance the deletion bitmap calculation for full duplicate load [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #22566: [enhancement](MOW) refactor delete bitmap calculator and enhance the deletion bitmap calculation for full duplicate load
URL: https://github.com/apache/doris/pull/22566
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org