You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "bobhan1 (via GitHub)" <gi...@apache.org> on 2024/10/11 07:12:13 UTC
[PR] [Fix](flexible partial update) Fix incorrect result when there are insert after delete for same key in one load [doris]
bobhan1 opened a new pull request, #41701:
URL: https://github.com/apache/doris/pull/41701
(no comment)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix incorrect result when there are insert after delete for same key in one load [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2443946016
<details>
<summary>TPC-DS: <b>Total hot run time: 197203 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d978d67bc61473d4e3aeadc3ec15eff34e8f7593, data reload: false
query1 1326 968 1017 968
query2 6245 2118 2090 2090
query3 11619 5000 4852 4852
query4 33467 23728 23653 23653
query5 4782 444 435 435
query6 276 172 163 163
query7 3995 283 293 283
query8 279 234 238 234
query9 9432 2630 2628 2628
query10 479 249 243 243
query11 18277 15193 15386 15193
query12 148 104 97 97
query13 1567 424 408 408
query14 9808 7089 7110 7089
query15 250 183 182 182
query16 8046 454 511 454
query17 1599 594 572 572
query18 2116 304 315 304
query19 220 154 163 154
query20 121 118 113 113
query21 208 108 104 104
query22 4611 4720 4191 4191
query23 34740 34379 34176 34176
query24 10960 2820 2857 2820
query25 627 418 423 418
query26 1443 159 162 159
query27 2623 277 296 277
query28 7646 2423 2396 2396
query29 933 431 417 417
query30 257 164 163 163
query31 1076 804 806 804
query32 91 59 54 54
query33 766 273 269 269
query34 953 519 517 517
query35 1034 897 922 897
query36 1109 954 965 954
query37 135 74 74 74
query38 4317 4229 4266 4229
query39 1523 1420 1410 1410
query40 260 99 99 99
query41 50 49 45 45
query42 110 100 101 100
query43 554 488 503 488
query44 1233 812 805 805
query45 177 165 170 165
query46 1135 706 718 706
query47 1929 1833 1865 1833
query48 411 307 317 307
query49 1015 404 400 400
query50 797 399 387 387
query51 7184 7047 6927 6927
query52 100 88 89 88
query53 256 184 182 182
query54 1090 408 407 407
query55 79 80 76 76
query56 268 267 239 239
query57 1302 1157 1153 1153
query58 222 200 233 200
query59 3235 3253 3021 3021
query60 265 257 251 251
query61 106 96 103 96
query62 843 674 662 662
query63 210 187 185 185
query64 4669 626 597 597
query65 3260 3208 3242 3208
query66 1056 301 296 296
query67 16081 15786 15669 15669
query68 4555 568 542 542
query69 436 254 253 253
query70 1193 1161 1148 1148
query71 314 261 250 250
query72 6166 4026 3920 3920
query73 759 351 360 351
query74 10493 8975 9055 8975
query75 3427 2697 2670 2670
query76 2650 973 974 973
query77 382 275 276 275
query78 10630 9582 9569 9569
query79 1247 592 600 592
query80 965 443 425 425
query81 552 243 244 243
query82 1180 125 115 115
query83 245 142 138 138
query84 226 71 68 68
query85 1407 309 281 281
query86 382 295 299 295
query87 4837 4766 4809 4766
query88 3416 2190 2163 2163
query89 414 288 290 288
query90 1981 187 186 186
query91 130 96 102 96
query92 58 50 47 47
query93 1263 550 538 538
query94 937 292 293 292
query95 348 244 245 244
query96 612 280 293 280
query97 2902 2726 2688 2688
query98 209 198 196 196
query99 1552 1323 1300 1300
Total cold run time: 302869 ms
Total hot run time: 197203 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix incorrect result when there are insert after delete for same key in one load [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring commented on code in PR #41701:
URL: https://github.com/apache/doris/pull/41701#discussion_r1818887847
##########
be/src/olap/partial_update_info.cpp:
##########
@@ -658,4 +664,422 @@ Status FlexibleReadPlan::fill_non_primary_key_columns_for_row_store(
return Status::OK();
}
+BlockAggregator::BlockAggregator(segment_v2::VerticalSegmentWriter& vertical_segment_writer)
+ : _writer(vertical_segment_writer), _tablet_schema(*_writer._tablet_schema) {}
+
+void BlockAggregator::merge_one_row(vectorized::MutableBlock& dst_block,
+ vectorized::Block* src_block, int64_t rid,
+ BitmapValue& skip_bitmap) {
+ for (size_t cid {_tablet_schema.num_key_columns()}; cid < _tablet_schema.num_columns(); cid++) {
+ if (cid == _tablet_schema.skip_bitmap_col_idx()) {
+ auto& cur_skip_bitmap =
+ assert_cast<vectorized::ColumnBitmap*>(dst_block.mutable_columns()[cid].get())
+ ->get_data()
+ .back();
+ const auto& new_row_skip_bitmap =
+ assert_cast<vectorized::ColumnBitmap*>(
+ src_block->get_by_position(cid).column->assume_mutable().get())
+ ->get_data()[rid];
+ cur_skip_bitmap &= new_row_skip_bitmap;
+ continue;
+ }
+ if (!skip_bitmap.contains(_tablet_schema.column(cid).unique_id())) {
+ dst_block.mutable_columns()[cid]->pop_back(1);
+ dst_block.mutable_columns()[cid]->insert_from(*src_block->get_by_position(cid).column,
+ rid);
+ }
+ }
+ VLOG_DEBUG << fmt::format("merge a row, after merge, output_block.rows()={}, state: {}",
+ dst_block.rows(), _state.to_string());
+}
+
+void BlockAggregator::append_one_row(vectorized::MutableBlock& dst_block,
+ vectorized::Block* src_block, int64_t rid) {
+ dst_block.add_row(src_block, rid);
+ _state.rows++;
+ VLOG_DEBUG << fmt::format("append a new row, after append, output_block.rows()={}, state: {}",
+ dst_block.rows(), _state.to_string());
+}
+
+void BlockAggregator::remove_last_n_rows(vectorized::MutableBlock& dst_block, int n) {
+ if (n > 0) {
+ for (size_t cid {0}; cid < _tablet_schema.num_columns(); cid++) {
+ DCHECK_GE(dst_block.mutable_columns()[cid]->size(), n);
+ dst_block.mutable_columns()[cid]->pop_back(n);
+ }
+ }
+}
+
+void BlockAggregator::append_or_merge_row(vectorized::MutableBlock& dst_block,
+ vectorized::Block* src_block, int64_t rid,
+ BitmapValue& skip_bitmap, bool have_delete_sign) {
+ if (have_delete_sign) {
+ // remove all the previous batched rows
+ remove_last_n_rows(dst_block, _state.rows);
+ _state.rows = 0;
+ _state.has_row_with_delete_sign = true;
+
+ append_one_row(dst_block, src_block, rid);
+ } else {
+ if (_state.should_merge()) {
+ merge_one_row(dst_block, src_block, rid, skip_bitmap);
+ } else {
+ append_one_row(dst_block, src_block, rid);
+ }
+ }
+};
+
+Status BlockAggregator::aggregate_rows(
+ vectorized::MutableBlock& output_block, vectorized::Block* block, int64_t start,
+ int64_t end, std::string key, std::vector<BitmapValue>* skip_bitmaps,
+ const signed char* delete_signs, vectorized::IOlapColumnDataAccessor* seq_column,
+ const std::vector<RowsetSharedPtr>& specified_rowsets,
+ std::vector<std::unique_ptr<SegmentCacheHandle>>& segment_caches) {
+ VLOG_DEBUG << fmt::format("merge rows in range=[{}-{})", start, end);
+ if (end - start == 1) {
+ output_block.add_row(block, start);
+ VLOG_DEBUG << fmt::format("append a row directly, rid={}", start);
+ return Status::OK();
+ }
+
+ auto seq_col_unique_id = _tablet_schema.column(_tablet_schema.sequence_col_idx()).unique_id();
+ auto delete_sign_col_unique_id =
+ _tablet_schema.column(_tablet_schema.delete_sign_idx()).unique_id();
+
+ _state.reset();
+
+ RowLocation loc;
+ RowsetSharedPtr rowset;
+ std::string previous_encoded_seq_value {};
+ Status st = _writer._tablet->lookup_row_key(
+ key, &_tablet_schema, false, specified_rowsets, &loc, _writer._mow_context->max_version,
+ segment_caches, &rowset, true, &previous_encoded_seq_value);
+ int64_t pos = start;
+ DCHECK(st.is<ErrorCode::KEY_NOT_FOUND>() || st.ok());
+
+ std::string cur_seq_val;
+ if (st.ok()) {
+ for (int64_t i {start}; i < end; i++) {
+ auto& skip_bitmap = skip_bitmaps->at(i);
+ bool row_has_sequence_col = (!skip_bitmap.contains(seq_col_unique_id));
+ // Find the first row that has larger sequence value than the existing row's
+ // or the first row that doesn't specify the sequence column.
+ // Discard all the rows before and begin to do aggregation from that row
Review Comment:
// Discard all the rows whose seq is little than previous_encoded_seq_value.
if (row_has_sequence_col) {
std::string seq_val {};
_writer._encode_seq_column(seq_column, i, &seq_val);
if (Slice {seq_val}.compare(Slice {previous_encoded_seq_value}) < 0) {
continue;
}
cur_seq_val = move(seq_val);
pos = i;
break;
}
cur_seq_val = previous;
pos = i;
break;
##########
be/src/olap/partial_update_info.cpp:
##########
@@ -658,4 +664,422 @@ Status FlexibleReadPlan::fill_non_primary_key_columns_for_row_store(
return Status::OK();
}
+BlockAggregator::BlockAggregator(segment_v2::VerticalSegmentWriter& vertical_segment_writer)
+ : _writer(vertical_segment_writer), _tablet_schema(*_writer._tablet_schema) {}
+
+void BlockAggregator::merge_one_row(vectorized::MutableBlock& dst_block,
+ vectorized::Block* src_block, int64_t rid,
+ BitmapValue& skip_bitmap) {
+ for (size_t cid {_tablet_schema.num_key_columns()}; cid < _tablet_schema.num_columns(); cid++) {
+ if (cid == _tablet_schema.skip_bitmap_col_idx()) {
+ auto& cur_skip_bitmap =
+ assert_cast<vectorized::ColumnBitmap*>(dst_block.mutable_columns()[cid].get())
+ ->get_data()
+ .back();
+ const auto& new_row_skip_bitmap =
+ assert_cast<vectorized::ColumnBitmap*>(
+ src_block->get_by_position(cid).column->assume_mutable().get())
+ ->get_data()[rid];
+ cur_skip_bitmap &= new_row_skip_bitmap;
+ continue;
+ }
+ if (!skip_bitmap.contains(_tablet_schema.column(cid).unique_id())) {
+ dst_block.mutable_columns()[cid]->pop_back(1);
+ dst_block.mutable_columns()[cid]->insert_from(*src_block->get_by_position(cid).column,
+ rid);
+ }
+ }
+ VLOG_DEBUG << fmt::format("merge a row, after merge, output_block.rows()={}, state: {}",
+ dst_block.rows(), _state.to_string());
+}
+
+void BlockAggregator::append_one_row(vectorized::MutableBlock& dst_block,
+ vectorized::Block* src_block, int64_t rid) {
+ dst_block.add_row(src_block, rid);
+ _state.rows++;
+ VLOG_DEBUG << fmt::format("append a new row, after append, output_block.rows()={}, state: {}",
+ dst_block.rows(), _state.to_string());
+}
+
+void BlockAggregator::remove_last_n_rows(vectorized::MutableBlock& dst_block, int n) {
+ if (n > 0) {
+ for (size_t cid {0}; cid < _tablet_schema.num_columns(); cid++) {
+ DCHECK_GE(dst_block.mutable_columns()[cid]->size(), n);
+ dst_block.mutable_columns()[cid]->pop_back(n);
+ }
+ }
+}
+
+void BlockAggregator::append_or_merge_row(vectorized::MutableBlock& dst_block,
+ vectorized::Block* src_block, int64_t rid,
+ BitmapValue& skip_bitmap, bool have_delete_sign) {
+ if (have_delete_sign) {
+ // remove all the previous batched rows
+ remove_last_n_rows(dst_block, _state.rows);
+ _state.rows = 0;
+ _state.has_row_with_delete_sign = true;
+
+ append_one_row(dst_block, src_block, rid);
+ } else {
+ if (_state.should_merge()) {
+ merge_one_row(dst_block, src_block, rid, skip_bitmap);
+ } else {
+ append_one_row(dst_block, src_block, rid);
+ }
+ }
+};
+
+Status BlockAggregator::aggregate_rows(
+ vectorized::MutableBlock& output_block, vectorized::Block* block, int64_t start,
+ int64_t end, std::string key, std::vector<BitmapValue>* skip_bitmaps,
+ const signed char* delete_signs, vectorized::IOlapColumnDataAccessor* seq_column,
+ const std::vector<RowsetSharedPtr>& specified_rowsets,
+ std::vector<std::unique_ptr<SegmentCacheHandle>>& segment_caches) {
+ VLOG_DEBUG << fmt::format("merge rows in range=[{}-{})", start, end);
+ if (end - start == 1) {
+ output_block.add_row(block, start);
+ VLOG_DEBUG << fmt::format("append a row directly, rid={}", start);
+ return Status::OK();
+ }
+
+ auto seq_col_unique_id = _tablet_schema.column(_tablet_schema.sequence_col_idx()).unique_id();
+ auto delete_sign_col_unique_id =
+ _tablet_schema.column(_tablet_schema.delete_sign_idx()).unique_id();
+
+ _state.reset();
+
+ RowLocation loc;
+ RowsetSharedPtr rowset;
+ std::string previous_encoded_seq_value {};
+ Status st = _writer._tablet->lookup_row_key(
+ key, &_tablet_schema, false, specified_rowsets, &loc, _writer._mow_context->max_version,
+ segment_caches, &rowset, true, &previous_encoded_seq_value);
+ int64_t pos = start;
+ DCHECK(st.is<ErrorCode::KEY_NOT_FOUND>() || st.ok());
+
+ std::string cur_seq_val;
+ if (st.ok()) {
+ for (int64_t i {start}; i < end; i++) {
+ auto& skip_bitmap = skip_bitmaps->at(i);
+ bool row_has_sequence_col = (!skip_bitmap.contains(seq_col_unique_id));
+ // Find the first row that has larger sequence value than the existing row's
+ // or the first row that doesn't specify the sequence column.
+ // Discard all the rows before and begin to do aggregation from that row
+ if (!row_has_sequence_col) {
+ cur_seq_val = std::move(previous_encoded_seq_value);
+ pos = i;
+ break;
+ } else {
+ std::string seq_val {};
+ _writer._encode_seq_column(seq_column, i, &seq_val);
+ if (Slice {seq_val}.compare(Slice {previous_encoded_seq_value}) >= 0) {
+ cur_seq_val = std::move(seq_val);
+ pos = i;
+ break;
+ }
+ }
+ }
+ } else {
+ pos = start;
+ auto& skip_bitmap = skip_bitmaps->at(pos);
+ bool row_has_sequence_col = (!skip_bitmap.contains(seq_col_unique_id));
+ if (row_has_sequence_col) {
+ std::string seq_val {};
+ // for rows that don't specify seqeunce col, seq_val will be encoded to minial value
+ _writer._encode_seq_column(seq_column, pos, &seq_val);
+ cur_seq_val = std::move(seq_val);
+ } else {
+ cur_seq_val.clear();
+ RETURN_IF_ERROR(_writer._generate_encoded_default_seq_value(
+ _tablet_schema, *_writer._opts.rowset_ctx->partial_update_info, &cur_seq_val));
+ }
+ }
+ VLOG_DEBUG << fmt::format("start pos={}, output_block.rows()={}", pos, output_block.rows());
+
+ for (int64_t rid {pos}; rid < end; rid++) {
+ auto& skip_bitmap = skip_bitmaps->at(rid);
+ bool row_has_sequence_col = (!skip_bitmap.contains(seq_col_unique_id));
+ bool have_delete_sign =
+ (!skip_bitmap.contains(delete_sign_col_unique_id) && delete_signs[rid] != 0);
+ if (!row_has_sequence_col) {
+ append_or_merge_row(output_block, block, rid, skip_bitmap, have_delete_sign);
+ } else {
+ std::string seq_val {};
+ _writer._encode_seq_column(seq_column, rid, &seq_val);
+ if (Slice {seq_val}.compare(Slice {cur_seq_val}) >= 0) {
+ append_or_merge_row(output_block, block, rid, skip_bitmap, have_delete_sign);
+ cur_seq_val = std::move(seq_val);
+ } else {
+ VLOG_DEBUG << fmt::format(
+ "skip rid={} becasue its seq value is lower than the previous", rid);
+ }
+ }
+ }
+ return Status::OK();
+};
+
+Status BlockAggregator::aggregate_for_sequence_column(
+ vectorized::Block* block, size_t num_rows,
+ const std::vector<vectorized::IOlapColumnDataAccessor*>& key_columns,
+ vectorized::IOlapColumnDataAccessor* seq_column,
+ const std::vector<RowsetSharedPtr>& specified_rowsets,
+ std::vector<std::unique_ptr<SegmentCacheHandle>>& segment_caches) {
+ DCHECK_EQ(block->columns(), _tablet_schema.num_columns());
+ VLOG_DEBUG << fmt::format("BlockAggregator::aggregate_for_sequence_column enter: block:{}\n",
+ block->dump_data());
+ // the process logic here is the same as MemTable::_aggregate_for_flexible_partial_update_without_seq_col()
+ // after this function, there will be at most 2 rows for a specified key
+ std::vector<BitmapValue>* skip_bitmaps =
+ &(assert_cast<vectorized::ColumnBitmap*>(
+ block->get_by_position(_tablet_schema.skip_bitmap_col_idx())
+ .column->assume_mutable()
+ .get())
+ ->get_data());
+ const auto* delete_signs = BaseTablet::get_delete_sign_column_data(*block, num_rows);
+
+ auto filtered_block = _tablet_schema.create_block();
+ vectorized::MutableBlock output_block =
+ vectorized::MutableBlock::build_mutable_block(&filtered_block);
+
+ int same_key_rows {0};
+ std::string previous_key {};
+ for (size_t block_pos {0}; block_pos < num_rows; block_pos++) {
+ std::string key = _writer._full_encode_keys(key_columns, block_pos);
+ if (block_pos > 0 && previous_key == key) {
+ same_key_rows++;
+ } else {
+ if (same_key_rows > 0) {
+ RETURN_IF_ERROR(aggregate_rows(output_block, block, block_pos - same_key_rows,
+ block_pos, std::move(previous_key), skip_bitmaps,
+ delete_signs, seq_column, specified_rowsets,
+ segment_caches));
+ }
+ same_key_rows = 1;
+ }
+ previous_key = std::move(key);
+ }
+ if (same_key_rows > 0) {
+ RETURN_IF_ERROR(aggregate_rows(output_block, block, num_rows - same_key_rows, num_rows,
+ std::move(previous_key), skip_bitmaps, delete_signs,
+ seq_column, specified_rowsets, segment_caches));
+ }
+
+ block->swap(output_block.to_block());
+ VLOG_DEBUG << fmt::format(
+ "BlockAggregator::aggregate_for_sequence_column after: data.block:{}\n",
+ block->dump_data());
+ return Status::OK();
+}
+
+Status BlockAggregator::fill_sequence_column(vectorized::Block* block, size_t num_rows,
+ const FixedReadPlan& read_plan,
+ std::vector<BitmapValue>& skip_bitmaps) {
+ DCHECK(_tablet_schema.has_sequence_col());
+ std::vector<uint32_t> cids {static_cast<uint32_t>(_tablet_schema.sequence_col_idx())};
+ auto seq_col_unique_id = _tablet_schema.column(_tablet_schema.sequence_col_idx()).unique_id();
+
+ auto seq_col_block = _tablet_schema.create_block_by_cids(cids);
+ auto tmp_block = _tablet_schema.create_block_by_cids(cids);
+ std::map<uint32_t, uint32_t> read_index;
+ RETURN_IF_ERROR(read_plan.read_columns_by_plan(_tablet_schema, cids, _writer._rsid_to_rowset,
+ seq_col_block, &read_index));
+
+ auto new_seq_col_ptr = tmp_block.get_by_position(0).column->assume_mutable();
+ const auto& old_seq_col_ptr = *seq_col_block.get_by_position(0).column;
+ const auto& cur_seq_col_ptr = *block->get_by_position(_tablet_schema.sequence_col_idx()).column;
+ for (size_t block_pos {0}; block_pos < num_rows; block_pos++) {
+ if (read_index.contains(block_pos)) {
+ new_seq_col_ptr->insert_from(old_seq_col_ptr, read_index[block_pos]);
+ skip_bitmaps[block_pos].remove(seq_col_unique_id);
+ } else {
+ new_seq_col_ptr->insert_from(cur_seq_col_ptr, block_pos);
+ }
+ }
+ block->replace_by_position(_tablet_schema.sequence_col_idx(), std::move(new_seq_col_ptr));
+ return Status::OK();
+}
+
+Status BlockAggregator::aggregate_for_insert_after_delete(
+ vectorized::Block* block, size_t num_rows,
+ const std::vector<vectorized::IOlapColumnDataAccessor*>& key_columns,
+ const std::vector<RowsetSharedPtr>& specified_rowsets,
+ std::vector<std::unique_ptr<SegmentCacheHandle>>& segment_caches) {
+ DCHECK_EQ(block->columns(), _tablet_schema.num_columns());
+ VLOG_DEBUG << fmt::format(
+ "BlockAggregator::aggregate_for_insert_after_delete enter: data.block:{}\n",
+ block->dump_data());
+ // there will be at most 2 rows for a specified key in block when control flow reaches here
+ // after this function, there will not be duplicate rows in block
+
+ std::vector<BitmapValue>* skip_bitmaps =
+ &(assert_cast<vectorized::ColumnBitmap*>(
+ block->get_by_position(_tablet_schema.skip_bitmap_col_idx())
+ .column->assume_mutable()
+ .get())
+ ->get_data());
+ const auto* delete_signs = BaseTablet::get_delete_sign_column_data(*block, num_rows);
+
+ auto filter_column = vectorized::ColumnUInt8::create(num_rows, 1);
+ auto* __restrict filter_map = filter_column->get_data().data();
+ std::string previous_key {};
+ bool previous_has_delete_sign {false};
+ int duplicate_rows {0};
+ int32_t delete_sign_col_unique_id =
+ _tablet_schema.column(_tablet_schema.delete_sign_idx()).unique_id();
+ auto seq_col_unique_id =
+ (_tablet_schema.sequence_col_idx() != -1)
+ ? _tablet_schema.column(_tablet_schema.sequence_col_idx()).unique_id()
+ : -1;
+ FixedReadPlan read_plan;
+ for (size_t block_pos {0}; block_pos < num_rows; block_pos++) {
+ size_t delta_pos = block_pos;
+ auto& skip_bitmap = skip_bitmaps->at(block_pos);
+ std::string key = _writer._full_encode_keys(key_columns, delta_pos);
+ bool have_delete_sign =
+ (!skip_bitmap.contains(delete_sign_col_unique_id) && delete_signs[block_pos] != 0);
+ if (delta_pos > 0 && previous_key == key) {
+ // !!ATTENTION!!: We can only remove the row with delete sign if there is a insert with the same key after this row.
+ // If there is only a row with delete sign, we should keep it and can't remove it from block, because
+ // compaction will not use the delete bitmap when reading data. So there may still be rows with delete sign
+ // in later process
+ DCHECK(previous_has_delete_sign);
+ DCHECK(!have_delete_sign);
+ ++duplicate_rows;
+ RowLocation loc;
+ RowsetSharedPtr rowset;
+ Status st = _writer._tablet->lookup_row_key(
+ key, &_tablet_schema, false, specified_rowsets, &loc,
+ _writer._mow_context->max_version, segment_caches, &rowset, true);
+ DCHECK(st.is<ErrorCode::KEY_NOT_FOUND>() || st.ok());
Review Comment:
check st.
##########
be/src/olap/partial_update_info.cpp:
##########
@@ -658,4 +664,422 @@ Status FlexibleReadPlan::fill_non_primary_key_columns_for_row_store(
return Status::OK();
}
+BlockAggregator::BlockAggregator(segment_v2::VerticalSegmentWriter& vertical_segment_writer)
+ : _writer(vertical_segment_writer), _tablet_schema(*_writer._tablet_schema) {}
+
+void BlockAggregator::merge_one_row(vectorized::MutableBlock& dst_block,
+ vectorized::Block* src_block, int64_t rid,
+ BitmapValue& skip_bitmap) {
+ for (size_t cid {_tablet_schema.num_key_columns()}; cid < _tablet_schema.num_columns(); cid++) {
+ if (cid == _tablet_schema.skip_bitmap_col_idx()) {
+ auto& cur_skip_bitmap =
+ assert_cast<vectorized::ColumnBitmap*>(dst_block.mutable_columns()[cid].get())
+ ->get_data()
+ .back();
+ const auto& new_row_skip_bitmap =
+ assert_cast<vectorized::ColumnBitmap*>(
+ src_block->get_by_position(cid).column->assume_mutable().get())
+ ->get_data()[rid];
+ cur_skip_bitmap &= new_row_skip_bitmap;
+ continue;
+ }
+ if (!skip_bitmap.contains(_tablet_schema.column(cid).unique_id())) {
+ dst_block.mutable_columns()[cid]->pop_back(1);
+ dst_block.mutable_columns()[cid]->insert_from(*src_block->get_by_position(cid).column,
+ rid);
+ }
+ }
+ VLOG_DEBUG << fmt::format("merge a row, after merge, output_block.rows()={}, state: {}",
+ dst_block.rows(), _state.to_string());
+}
+
+void BlockAggregator::append_one_row(vectorized::MutableBlock& dst_block,
+ vectorized::Block* src_block, int64_t rid) {
+ dst_block.add_row(src_block, rid);
+ _state.rows++;
+ VLOG_DEBUG << fmt::format("append a new row, after append, output_block.rows()={}, state: {}",
+ dst_block.rows(), _state.to_string());
+}
+
+void BlockAggregator::remove_last_n_rows(vectorized::MutableBlock& dst_block, int n) {
+ if (n > 0) {
+ for (size_t cid {0}; cid < _tablet_schema.num_columns(); cid++) {
+ DCHECK_GE(dst_block.mutable_columns()[cid]->size(), n);
+ dst_block.mutable_columns()[cid]->pop_back(n);
+ }
+ }
+}
+
+void BlockAggregator::append_or_merge_row(vectorized::MutableBlock& dst_block,
+ vectorized::Block* src_block, int64_t rid,
+ BitmapValue& skip_bitmap, bool have_delete_sign) {
+ if (have_delete_sign) {
+ // remove all the previous batched rows
+ remove_last_n_rows(dst_block, _state.rows);
+ _state.rows = 0;
+ _state.has_row_with_delete_sign = true;
+
+ append_one_row(dst_block, src_block, rid);
+ } else {
+ if (_state.should_merge()) {
+ merge_one_row(dst_block, src_block, rid, skip_bitmap);
+ } else {
+ append_one_row(dst_block, src_block, rid);
+ }
+ }
+};
+
+Status BlockAggregator::aggregate_rows(
+ vectorized::MutableBlock& output_block, vectorized::Block* block, int64_t start,
+ int64_t end, std::string key, std::vector<BitmapValue>* skip_bitmaps,
+ const signed char* delete_signs, vectorized::IOlapColumnDataAccessor* seq_column,
+ const std::vector<RowsetSharedPtr>& specified_rowsets,
+ std::vector<std::unique_ptr<SegmentCacheHandle>>& segment_caches) {
+ VLOG_DEBUG << fmt::format("merge rows in range=[{}-{})", start, end);
+ if (end - start == 1) {
+ output_block.add_row(block, start);
+ VLOG_DEBUG << fmt::format("append a row directly, rid={}", start);
+ return Status::OK();
+ }
+
+ auto seq_col_unique_id = _tablet_schema.column(_tablet_schema.sequence_col_idx()).unique_id();
+ auto delete_sign_col_unique_id =
+ _tablet_schema.column(_tablet_schema.delete_sign_idx()).unique_id();
+
+ _state.reset();
+
+ RowLocation loc;
+ RowsetSharedPtr rowset;
+ std::string previous_encoded_seq_value {};
+ Status st = _writer._tablet->lookup_row_key(
+ key, &_tablet_schema, false, specified_rowsets, &loc, _writer._mow_context->max_version,
+ segment_caches, &rowset, true, &previous_encoded_seq_value);
+ int64_t pos = start;
+ DCHECK(st.is<ErrorCode::KEY_NOT_FOUND>() || st.ok());
Review Comment:
There are other errors can be reported by lookup_row_key, we should handle it. DCHECK is not enough.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2464366572
TeamCity be ut coverage result:
Function Coverage: 37.88% (9867/26045)
Line Coverage: 29.07% (82283/283063)
Region Coverage: 28.20% (42339/150115)
Branch Coverage: 24.76% (21440/86602)
Coverage Report: http://coverage.selectdb-in.cc/coverage/d99a063f469f8323415222d10da3c57f6ed0ff38_d99a063f469f8323415222d10da3c57f6ed0ff38/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix incorrect result when there are insert after delete for same key in one load [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #41701:
URL: https://github.com/apache/doris/pull/41701#discussion_r1804442705
##########
be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:
##########
@@ -831,98 +885,249 @@ Status VerticalSegmentWriter::_generate_flexible_read_plan(
Status VerticalSegmentWriter::_merge_rows_for_sequence_column(
Review Comment:
warning: function '_merge_rows_for_sequence_column' has cognitive complexity of 76 (threshold 50) [readability-function-cognitive-complexity]
```cpp
Status VerticalSegmentWriter::_merge_rows_for_sequence_column(
^
```
<details>
<summary>Additional context</summary>
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:905:** nesting level increased to 1
```cpp
auto append_row = [&](int64_t rid, BitmapValue& skip_bitmap, bool have_delete_sign) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:906:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (have_delete_sign) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:919:** +1, nesting level increased to 2
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:920:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if ((cur_rows == 1 && !have_rows_with_delete_sign) || cur_rows == 2) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:920:** +1
```cpp
if ((cur_rows == 1 && !have_rows_with_delete_sign) || cur_rows == 2) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:920:** +1
```cpp
if ((cur_rows == 1 && !have_rows_with_delete_sign) || cur_rows == 2) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:943:** +1, nesting level increased to 3
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:954:** nesting level increased to 1
```cpp
auto aggregate_rows = [&](std::string key, int64_t start, int64_t end) -> Status {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:956:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (end - start == 1) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:971:** +1
```cpp
DCHECK(st.is<KEY_NOT_FOUND>() || st.ok());
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:974:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (st.ok()) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:975:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
for (int64_t i {start}; i < end; i++) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:981:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
if (!row_has_sequence_col) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:986:** +1, nesting level increased to 4
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:989:** +5, including nesting penalty of 4, nesting level increased to 5
```cpp
if (Slice {seq_val}.compare(Slice {previous_encoded_seq_value}) >= 0) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:997:** +1, nesting level increased to 2
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1001:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (row_has_sequence_col) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1007:** +1, nesting level increased to 3
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1009:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
RETURN_IF_ERROR(_generate_encoded_default_seq_value(
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1009:** +5, including nesting penalty of 4, nesting level increased to 5
```cpp
RETURN_IF_ERROR(_generate_encoded_default_seq_value(
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1016:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
for (int64_t rid {pos + 1}; rid < end; rid++) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1020:** +1
```cpp
(!skip_bitmap.contains(delete_sign_col_unique_id) && delete_signs[rid] != 0);
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1021:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (!row_has_sequence_col) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1023:** +1, nesting level increased to 3
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1026:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
if (Slice {seq_val}.compare(Slice {cur_seq_val}) >= 0) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1029:** +1, nesting level increased to 4
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1040:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
for (size_t block_pos {0}; block_pos < data.num_rows; block_pos++) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1042:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (block_pos > 0 && previous_key == key) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1042:** +1
```cpp
if (block_pos > 0 && previous_key == key) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1044:** +1, nesting level increased to 2
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1045:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (same_key_rows > 0) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1046:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
RETURN_IF_ERROR(aggregate_rows(std::move(previous_key), block_pos - same_key_rows,
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1046:** +5, including nesting penalty of 4, nesting level increased to 5
```cpp
RETURN_IF_ERROR(aggregate_rows(std::move(previous_key), block_pos - same_key_rows,
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1053:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
if (same_key_rows > 0) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1054:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(aggregate_rows(std::move(previous_key), data.num_rows - same_key_rows,
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1054:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(aggregate_rows(std::move(previous_key), data.num_rows - same_key_rows,
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix incorrect result when there are insert after delete for same key in one load [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #41701:
URL: https://github.com/apache/doris/pull/41701#discussion_r1797631570
##########
be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:
##########
@@ -831,15 +885,18 @@ Status VerticalSegmentWriter::_generate_flexible_read_plan(
Status VerticalSegmentWriter::_merge_rows_for_sequence_column(
Review Comment:
warning: function '_merge_rows_for_sequence_column' has cognitive complexity of 76 (threshold 50) [readability-function-cognitive-complexity]
```cpp
Status VerticalSegmentWriter::_merge_rows_for_sequence_column(
^
```
<details>
<summary>Additional context</summary>
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:905:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
RETURN_IF_ERROR(_generate_encoded_default_seq_value(
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:905:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(_generate_encoded_default_seq_value(
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:916:** nesting level increased to 1
```cpp
auto get_idx = [](bool with_seq_col, bool has_delete_sign) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:917:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (!with_seq_col) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:918:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (has_delete_sign) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:920:** +1, nesting level increased to 3
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:923:** +1, nesting level increased to 2
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:924:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (has_delete_sign) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:926:** +1, nesting level increased to 3
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:932:** nesting level increased to 1
```cpp
auto find_rows_to_filter = [&](const std::string& key) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:935:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (has_row_with_seq_col && has_row_without_seq_col) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:935:** +1
```cpp
if (has_row_with_seq_col && has_row_without_seq_col) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:945:** +1
```cpp
DCHECK(st.is<KEY_NOT_FOUND>() || st.ok());
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:947:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (st.is<KEY_NOT_FOUND>()) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:949:** +1, nesting level increased to 3
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:954:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (batched_rows[2] != -1) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:958:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (batched_rows[3] != -1) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:963:** nesting level increased to 3
```cpp
auto remove_rows_without_seq = [&]() {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:964:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
if (batched_rows[0] != -1) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:968:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
if (batched_rows[1] != -1) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:975:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (batched_rows[2] != -1 && batched_rows[3] != -1) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:976:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
if (previous_seq_slice.compare(Slice {row_with_delete_sign_encoded_seq_value}) <=
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:979:** +1, nesting level increased to 4
```cpp
} else if (previous_seq_slice.compare(
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:984:** +1, nesting level increased to 4
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:990:** +1, nesting level increased to 3
```cpp
} else if (batched_rows[2] != -1) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:991:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
if (previous_seq_slice.compare(Slice {row_with_delete_sign_encoded_seq_value}) <=
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:994:** +1, nesting level increased to 4
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:998:** +1, nesting level increased to 3
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:999:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
if (previous_seq_slice.compare(Slice {row_without_delete_sign_encoded_seq_value}) <=
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1002:** +1, nesting level increased to 4
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1012:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
for (size_t block_pos = data.row_pos; block_pos < data.row_pos + data.num_rows; block_pos++) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1018:** +1
```cpp
(!skip_bitmap.contains(delete_sign_col_unique_id) && delete_signs[block_pos] != 0);
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1021:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (delta_pos > 0 && previous_key == key) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1021:** +1
```cpp
if (delta_pos > 0 && previous_key == key) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1022:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (!has_same_rows) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1028:** +1, nesting level increased to 2
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1029:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (has_same_rows) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1037:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
if (has_same_rows) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1040:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
if (duplicate_keys > 0) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1041:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(_filter_block_for_flexible_partial_update(
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1041:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(_filter_block_for_flexible_partial_update(
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix incorrect result when there are insert after delete for same key in one load [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2408550893
<details>
<summary>TPC-DS: <b>Total hot run time: 192006 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 02cb8af6bdf15fd2a892f3a48fa8c8b4446761a7, data reload: false
query1 921 404 388 388
query2 6258 2109 2109 2109
query3 8694 190 208 190
query4 34238 23596 23396 23396
query5 3520 478 474 474
query6 266 165 167 165
query7 4185 299 305 299
query8 287 249 212 212
query9 9532 2654 2645 2645
query10 416 281 284 281
query11 17678 15208 15604 15208
query12 152 97 93 93
query13 1569 459 450 450
query14 9381 7692 7240 7240
query15 242 168 169 168
query16 7296 440 433 433
query17 1667 624 595 595
query18 1393 319 320 319
query19 298 160 154 154
query20 121 113 115 113
query21 206 103 107 103
query22 4746 4496 4326 4326
query23 35012 34207 34207 34207
query24 11103 2856 2916 2856
query25 606 421 410 410
query26 1068 171 168 168
query27 2354 301 300 300
query28 7389 2415 2409 2409
query29 761 447 429 429
query30 259 159 156 156
query31 1058 800 794 794
query32 96 56 56 56
query33 765 299 303 299
query34 896 504 503 503
query35 877 750 728 728
query36 1084 945 951 945
query37 153 92 88 88
query38 3992 3835 3803 3803
query39 1474 1426 1434 1426
query40 206 98 97 97
query41 47 44 46 44
query42 125 102 97 97
query43 533 499 502 499
query44 1217 826 806 806
query45 193 163 168 163
query46 1175 714 726 714
query47 1942 1855 1837 1837
query48 443 364 349 349
query49 928 417 411 411
query50 821 428 423 423
query51 7133 6943 6961 6943
query52 103 91 86 86
query53 256 181 182 181
query54 1241 476 480 476
query55 83 78 75 75
query56 309 280 273 273
query57 1214 1152 1171 1152
query58 242 234 246 234
query59 3392 3107 3079 3079
query60 303 273 266 266
query61 103 99 102 99
query62 878 681 674 674
query63 220 194 190 190
query64 4340 641 610 610
query65 3249 3292 3208 3208
query66 1266 311 306 306
query67 16017 15554 15634 15554
query68 4427 567 535 535
query69 508 306 310 306
query70 1188 1111 1120 1111
query71 368 270 283 270
query72 7185 4027 3992 3992
query73 781 352 354 352
query74 10401 8950 8863 8863
query75 3392 2704 2724 2704
query76 3046 921 865 865
query77 573 298 286 286
query78 10437 9609 9527 9527
query79 3470 613 600 600
query80 2328 455 458 455
query81 587 237 232 232
query82 725 139 137 137
query83 322 129 132 129
query84 281 79 78 78
query85 1122 295 294 294
query86 324 302 306 302
query87 4513 4426 4268 4268
query88 4078 2396 2362 2362
query89 416 287 290 287
query90 2083 187 182 182
query91 151 107 104 104
query92 61 46 48 46
query93 1683 520 533 520
query94 886 306 298 298
query95 354 256 252 252
query96 619 276 273 273
query97 3245 3109 3098 3098
query98 216 217 190 190
query99 1510 1324 1304 1304
Total cold run time: 301272 ms
Total hot run time: 192006 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix incorrect result when there are insert after delete for same key in one load [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #41701:
URL: https://github.com/apache/doris/pull/41701#discussion_r1800999541
##########
be/src/olap/memtable.cpp:
##########
@@ -578,6 +539,200 @@ void MemTable::_aggregate() {
}
}
+template <bool is_final>
+void MemTable::_aggregate_for_flexible_partial_update_without_seq_col(
+ const vectorized::ColumnsWithTypeAndName& block_data,
+ vectorized::MutableBlock& mutable_block, std::vector<RowInBlock*>& temp_row_in_blocks) {
+ RowInBlock* prev_row = nullptr;
+ int row_pos = -1;
+ auto& skip_bitmaps = assert_cast<vectorized::ColumnBitmap*>(
+ mutable_block.mutable_columns()[_skip_bitmap_col_idx].get())
+ ->get_data();
+ auto& delete_signs = assert_cast<vectorized::ColumnInt8*>(
+ mutable_block.mutable_columns()[_delete_sign_col_idx].get())
+ ->get_data();
+ RowInBlock* row_with_delete_sign {nullptr};
+ RowInBlock* row_without_delete_sign {nullptr};
+
+ auto finalize_rows = [&]() {
+ if (row_with_delete_sign != nullptr) {
+ temp_row_in_blocks.push_back(row_with_delete_sign);
+ _finalize_one_row<is_final, false>(row_with_delete_sign, block_data, ++row_pos);
+ row_with_delete_sign = nullptr;
+ }
+ if (row_without_delete_sign != nullptr) {
+ temp_row_in_blocks.push_back(row_without_delete_sign);
+ _finalize_one_row<is_final, false>(row_without_delete_sign, block_data, ++row_pos);
+ row_without_delete_sign = nullptr;
+ }
+ _arena->clear();
+ };
+
+ auto add_row = [&](RowInBlock* row, bool with_delete_sign) {
+ if (with_delete_sign) {
+ row_with_delete_sign = row;
+ } else {
+ row_without_delete_sign = row;
+ }
+ };
+ for (RowInBlock* cur_row : _row_in_blocks) {
+ const BitmapValue& skip_bitmap = skip_bitmaps[cur_row->_row_pos];
+ bool cur_row_has_delete_sign = (!skip_bitmap.contains(_delete_sign_col_unique_id) &&
+ delete_signs[cur_row->_row_pos] != 0);
+ prev_row =
+ (row_with_delete_sign == nullptr) ? row_without_delete_sign : row_with_delete_sign;
+ // compare keys, the keys of row_with_delete_sign and row_without_delete_sign is the same,
+ // choose any of them if it's valid
+ if (prev_row != nullptr && (*_vec_row_comparator)(prev_row, cur_row) == 0) {
+ if (cur_row_has_delete_sign) {
+ if (row_without_delete_sign != nullptr) {
+ // if there exits row without delete sign, remove it first
+ _clear_row_agg(row_without_delete_sign);
+ _stat.merged_rows++;
+ row_without_delete_sign = nullptr;
+ }
+ // and then unconditionally replace the previous row
+ prev_row = row_with_delete_sign;
+ } else {
+ prev_row = row_without_delete_sign;
+ }
+
+ if (prev_row == nullptr) {
+ add_row(cur_row, cur_row_has_delete_sign);
+ } else {
+ if (!prev_row->has_init_agg()) {
+ _init_row_for_agg(prev_row, mutable_block);
+ }
+ _stat.merged_rows++;
+ _aggregate_two_row_in_block<true>(mutable_block, cur_row, prev_row);
+ }
+ } else {
+ finalize_rows();
+ add_row(cur_row, cur_row_has_delete_sign);
+ }
+ }
+ // finalize the last lows
+ finalize_rows();
+}
+
+template <bool is_final>
+void MemTable::_aggregate_for_flexible_partial_update_with_seq_col(
Review Comment:
warning: function '_aggregate_for_flexible_partial_update_with_seq_col' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
void MemTable::_aggregate_for_flexible_partial_update_with_seq_col(
^
```
<details>
<summary>Additional context</summary>
**be/src/olap/memtable.cpp:618:** 113 lines including whitespace and comments (threshold 80)
```cpp
void MemTable::_aggregate_for_flexible_partial_update_with_seq_col(
^
```
</details>
##########
be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:
##########
@@ -831,15 +885,20 @@ Status VerticalSegmentWriter::_generate_flexible_read_plan(
Status VerticalSegmentWriter::_merge_rows_for_sequence_column(
Review Comment:
warning: function '_merge_rows_for_sequence_column' has cognitive complexity of 74 (threshold 50) [readability-function-cognitive-complexity]
```cpp
Status VerticalSegmentWriter::_merge_rows_for_sequence_column(
^
```
<details>
<summary>Additional context</summary>
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:907:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
RETURN_IF_ERROR(_generate_encoded_default_seq_value(
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:907:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(_generate_encoded_default_seq_value(
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:918:** nesting level increased to 1
```cpp
auto get_idx = [](bool with_seq_col, bool has_delete_sign) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:919:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (!with_seq_col) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:920:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
return (has_delete_sign ? 0 : 1);
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:921:** +1, nesting level increased to 2
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:922:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
return (has_delete_sign ? 2 : 3);
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:926:** nesting level increased to 1
```cpp
auto find_rows_to_filter = [&](const std::string& key) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:929:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (has_row_with_seq_col && has_row_without_seq_col) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:929:** +1
```cpp
if (has_row_with_seq_col && has_row_without_seq_col) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:944:** +1
```cpp
DCHECK(st.is<KEY_NOT_FOUND>() || st.ok());
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:946:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (st.is<KEY_NOT_FOUND>()) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:948:** +1, nesting level increased to 3
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:953:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (batched_rows[2] != -1) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:957:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (batched_rows[3] != -1) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:962:** nesting level increased to 3
```cpp
auto remove_rows_without_seq = [&]() {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:963:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
if (batched_rows[0] != -1) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:967:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
if (batched_rows[1] != -1) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:974:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (batched_rows[2] != -1 && batched_rows[3] != -1) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:976:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
if (previous_seq_slice.compare(Slice {row_with_delete_sign_encoded_seq_value}) <=
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:979:** +1, nesting level increased to 4
```cpp
} else if (previous_seq_slice.compare(
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:984:** +1, nesting level increased to 4
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:990:** +1, nesting level increased to 3
```cpp
} else if (batched_rows[2] != -1) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:991:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
if (previous_seq_slice.compare(Slice {row_with_delete_sign_encoded_seq_value}) <=
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:994:** +1, nesting level increased to 4
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:998:** +1, nesting level increased to 3
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:999:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
if (previous_seq_slice.compare(Slice {row_without_delete_sign_encoded_seq_value}) <=
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1002:** +1, nesting level increased to 4
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1012:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
for (size_t block_pos = data.row_pos; block_pos < data.row_pos + data.num_rows; block_pos++) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1018:** +1
```cpp
(!skip_bitmap.contains(delete_sign_col_unique_id) && delete_signs[block_pos] != 0);
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1021:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (delta_pos > 0 && previous_key == key) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1021:** +1
```cpp
if (delta_pos > 0 && previous_key == key) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1022:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (!has_same_rows) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1028:** +1, nesting level increased to 2
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1029:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (has_same_rows) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1037:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
if (has_same_rows) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1040:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
if (duplicate_keys > 0) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1041:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(_filter_block_for_flexible_partial_update(
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1041:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(_filter_block_for_flexible_partial_update(
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix incorrect result when there are insert after delete for same key in one load [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring commented on code in PR #41701:
URL: https://github.com/apache/doris/pull/41701#discussion_r1802413848
##########
be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:
##########
@@ -883,45 +951,160 @@ Status VerticalSegmentWriter::_merge_rows_for_sequence_column(
_rsid_to_rowset.emplace(rowset->rowset_id(), rowset);
previous_seq_slice = Slice {previous_encoded_seq_value};
}
- std::string cur_encoded_seq_value {};
- _encode_seq_column(seq_column, rid_with_seq, &cur_encoded_seq_value);
+ if (batched_rows[2] != -1) {
+ _encode_seq_column(seq_column, batched_rows[2],
+ &row_with_delete_sign_encoded_seq_value);
+ }
+ if (batched_rows[3] != -1) {
+ _encode_seq_column(seq_column, batched_rows[3],
+ &row_without_delete_sign_encoded_seq_value);
+ }
+
+ auto remove_rows_without_seq = [&]() {
+ if (batched_rows[0] != -1) {
+ filter_map[batched_rows[0]] = 0;
+ ++duplicate_keys;
+ }
+ if (batched_rows[1] != -1) {
+ filter_map[batched_rows[1]] = 0;
+ ++duplicate_keys;
+ }
+ };
+
// the encoded value is order-preserving, so we can use Slice::compare() to compare them
- int res = previous_seq_slice.compare(Slice {cur_encoded_seq_value});
- VLOG_DEBUG << fmt::format(
- "VerticalSegmentWriter::_merge_rows_for_sequence_column: rid_with_seq={}, "
- "rid_missing_seq={}, res={}",
- rid_with_seq, rid_missing_seq, res);
- if (res > 0) {
- filter_map[rid_with_seq] = 0;
- } else if (res < 0) {
- filter_map[rid_missing_seq] = 0;
+ if (batched_rows[2] != -1 && batched_rows[3] != -1) {
+ // it's guaranteed that the sequence value of batched_rows[2]'s is strictly smaller than batched_rows[3]'s
+ if (previous_seq_slice.compare(Slice {row_with_delete_sign_encoded_seq_value}) <=
+ 0) {
+ remove_rows_without_seq();
+ } else if (previous_seq_slice.compare(
+ Slice {row_without_delete_sign_encoded_seq_value}) <= 0) {
+ remove_rows_without_seq();
+ filter_map[batched_rows[2]] = 0;
+ ++duplicate_keys;
+ } else {
+ filter_map[batched_rows[2]] = 0;
+ ++duplicate_keys;
+ filter_map[batched_rows[3]] = 0;
+ ++duplicate_keys;
+ }
+ } else if (batched_rows[2] != -1) {
+ if (previous_seq_slice.compare(Slice {row_with_delete_sign_encoded_seq_value}) <=
+ 0) {
+ remove_rows_without_seq();
+ } else {
+ filter_map[batched_rows[2]] = 0;
+ ++duplicate_keys;
+ }
} else {
- filter_map[std::min(rid_with_seq, rid_missing_seq)] = 0;
+ if (previous_seq_slice.compare(Slice {row_without_delete_sign_encoded_seq_value}) <=
+ 0) {
+ remove_rows_without_seq();
+ } else {
+ filter_map[batched_rows[3]] = 0;
+ ++duplicate_keys;
+ }
+ }
+ }
+ has_same_rows = false;
+ batched_rows.fill(-1);
+ };
+
+ for (size_t block_pos = data.row_pos; block_pos < data.row_pos + data.num_rows; block_pos++) {
+ size_t delta_pos = block_pos - data.row_pos;
+ auto& skip_bitmap = skip_bitmaps->at(block_pos);
+ std::string key = _full_encode_keys(key_columns, delta_pos);
+ bool row_has_sequence_col = (!skip_bitmap.contains(seq_col_unique_id));
+ bool row_has_delete_sign =
+ (!skip_bitmap.contains(delete_sign_col_unique_id) && delete_signs[block_pos] != 0);
+ VLOG_DEBUG << fmt::format("block_pos={}, row_has_sequence_col={}, row_has_delete_sign={}",
+ block_pos, row_has_sequence_col, row_has_delete_sign);
+ if (delta_pos > 0 && previous_key == key) {
+ if (!has_same_rows) {
+ batched_rows[get_idx(previous_has_seq_col, previous_has_delete_sign)] =
+ block_pos - 1;
+ has_same_rows = true;
+ }
+ batched_rows[get_idx(row_has_sequence_col, row_has_delete_sign)] = block_pos;
+ } else {
+ if (has_same_rows) {
+ find_rows_to_filter(previous_key);
}
}
previous_key = std::move(key);
previous_has_seq_col = row_has_sequence_col;
+ previous_has_delete_sign = row_has_delete_sign;
+ }
+ if (has_same_rows) {
+ find_rows_to_filter(previous_key);
}
if (duplicate_keys > 0) {
- auto num_cols = data.block->columns();
- auto* block = const_cast<vectorized::Block*>(data.block);
- block->insert({std::move(filter_column), std::make_shared<vectorized::DataTypeUInt8>(),
- "__dup_key_filter_col__"});
- RETURN_IF_ERROR(vectorized::Block::filter_block(block, num_cols, num_cols));
- int merged_rows = data.num_rows - block->rows();
- VLOG_DEBUG << fmt::format(
- "VerticalSegmentWriter::_merge_rows_for_sequence_column after filter: "
- "data.block:{}\n",
- data.block->dump_data());
- if (duplicate_keys != merged_rows) {
- auto msg = fmt::format(
- "duplicate_keys != merged_rows, duplicate_keys={}, merged_rows={}, "
- "num_rows={}, mutable_block->rows()={}",
- duplicate_keys, merged_rows, data.num_rows, block->rows());
- DCHECK(false) << msg;
- return Status::InternalError<false>(msg);
+ RETURN_IF_ERROR(_filter_block_for_flexible_partial_update(
+ data, std::move(filter_column), duplicate_keys, "__dup_key_filter_col__"));
+ }
+ return Status::OK();
+}
+
+Status VerticalSegmentWriter::_merge_rows_for_insert_after_delete(
+ RowsInBlock& data, std::vector<BitmapValue>* skip_bitmaps,
+ const std::vector<vectorized::IOlapColumnDataAccessor*>& key_columns,
+ vectorized::IOlapColumnDataAccessor* seq_column, const signed char* delete_signs,
+ const std::vector<RowsetSharedPtr>& specified_rowsets,
+ std::vector<std::unique_ptr<SegmentCacheHandle>>& segment_caches) {
+ VLOG_DEBUG << fmt::format(
+ "VerticalSegmentWriter::_merge_rows_for_insert_after_delete enter: data.block:{}\n",
+ data.block->dump_data());
+ // there will be at most 2 rows for a specified key in block when control flow reaches here
+ // after this function, there will not be duplicate rows in block
+ auto filter_column = vectorized::ColumnUInt8::create(data.num_rows, 1);
+ auto* __restrict filter_map = filter_column->get_data().data();
+ std::string previous_key {};
+ bool previous_has_delete_sign {false};
+ int duplicate_rows {0};
+ int32_t delete_sign_col_unique_id =
+ _tablet_schema->column(_tablet_schema->delete_sign_idx()).unique_id();
+ for (size_t block_pos = data.row_pos; block_pos < data.row_pos + data.num_rows; block_pos++) {
+ size_t delta_pos = block_pos - data.row_pos;
+ auto& skip_bitmap = skip_bitmaps->at(block_pos);
+ std::string key = _full_encode_keys(key_columns, delta_pos);
+ bool have_delete_sign =
+ (!skip_bitmap.contains(delete_sign_col_unique_id) && delete_signs[block_pos] != 0);
+ if (delta_pos > 0 && previous_key == key) {
+ // !!ATTENTION!!: We can only remove the row with delete sign if there is a insert with the same key after this row.
+ // If there is only a row with delete sign, we should keep it and can't remove it from block, because
+ // compaction will not use the delete bitmap when reading data. So there may still be rows with delete sign
+ // in later process
+ DCHECK(previous_has_delete_sign);
+ DCHECK(!have_delete_sign);
+ ++duplicate_rows;
+ RowLocation loc;
+ RowsetSharedPtr rowset;
+ Status st = _tablet->lookup_row_key(key, _tablet_schema.get(), false, specified_rowsets,
+ &loc, _mow_context->max_version, segment_caches,
+ &rowset, true);
+ DCHECK(st.is<KEY_NOT_FOUND>() || st.ok());
+
+ Slice previous_seq_slice {};
+ if (st.ok()) {
+ // delete the existing row
+ _mow_context->delete_bitmap->add(
+ {loc.rowset_id, loc.segment_id, DeleteBitmap::TEMP_VERSION_COMMON},
+ loc.row_id);
+ VLOG_DEBUG << fmt::format(
+ "_merge_rows_for_insert_after_delete: delete existing row, rowset_id={}, "
+ "segment_id={}, row_id={}",
+ loc.rowset_id.to_string(), loc.segment_id, loc.row_id);
Review Comment:
Maybe we should implement a new VerticalSegmentWriter inherts from VerticalSegmentWriter which allows duplicated rows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-3067595169
run cloud_p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-3059976085
# Cloud UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/9df5bf195299c9f1d859aae1c78853df8600e7d3_9df5bf195299c9f1d859aae1c78853df8600e7d3_cloud/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/9df5bf195299c9f1d859aae1c78853df8600e7d3_9df5bf195299c9f1d859aae1c78853df8600e7d3_cloud/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 82.05% (1257/1532) |
| Line Coverage | 67.85% (21500/31689) |
| Region Coverage | 67.42% (10651/15799) |
| Branch Coverage | 56.95% (5615/9860) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-3060574221
<details>
<summary>TPC-DS: <b>Total hot run time: 187397 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8139e2d8e542a5e69903ac8e087381bed76cc5fe, data reload: false
query1 1008 375 406 375
query2 6498 1722 1678 1678
query3 6740 212 223 212
query4 26424 23820 22996 22996
query5 4306 598 433 433
query6 314 213 208 208
query7 4640 502 309 309
query8 276 220 224 220
query9 8609 2647 2667 2647
query10 502 330 264 264
query11 15783 15238 14825 14825
query12 145 105 108 105
query13 1624 526 387 387
query14 9089 6115 6016 6016
query15 205 188 163 163
query16 7284 425 256 256
query17 1334 703 569 569
query18 1974 392 298 298
query19 199 191 160 160
query20 122 123 116 116
query21 206 120 103 103
query22 4083 4067 4034 4034
query23 33986 33307 33198 33198
query24 8493 2385 2423 2385
query25 550 454 398 398
query26 1231 268 149 149
query27 2763 502 338 338
query28 4345 2133 2131 2131
query29 826 569 457 457
query30 293 219 208 208
query31 924 839 758 758
query32 76 66 63 63
query33 537 334 279 279
query34 791 837 522 522
query35 578 615 564 564
query36 951 993 888 888
query37 114 95 79 79
query38 4070 4133 4135 4133
query39 1486 1442 1480 1442
query40 208 116 109 109
query41 58 54 53 53
query42 121 115 110 110
query43 505 491 470 470
query44 1339 826 839 826
query45 176 171 171 171
query46 846 1026 648 648
query47 1757 1769 1707 1707
query48 383 423 317 317
query49 751 491 374 374
query50 635 683 422 422
query51 5499 5590 5444 5444
query52 112 110 100 100
query53 228 265 186 186
query54 587 606 509 509
query55 83 83 87 83
query56 321 298 271 271
query57 1174 1174 1106 1106
query58 271 258 258 258
query59 2631 2699 2543 2543
query60 337 368 318 318
query61 124 123 132 123
query62 798 737 643 643
query63 228 190 187 187
query64 4377 1156 883 883
query65 4275 4198 4167 4167
query66 1134 409 309 309
query67 15676 15320 15347 15320
query68 4717 909 547 547
query69 518 315 268 268
query70 1144 1133 1122 1122
query71 414 324 322 322
query72 5837 4750 4801 4750
query73 674 617 356 356
query74 8964 9209 8947 8947
query75 3164 3216 2695 2695
query76 3198 1129 747 747
query77 521 396 296 296
query78 11125 11159 10391 10391
query79 991 827 589 589
query80 1003 549 451 451
query81 509 251 223 223
query82 330 123 96 96
query83 268 244 223 223
query84 242 106 83 83
query85 752 358 321 321
query86 330 306 286 286
query87 4458 4467 4254 4254
query88 2884 2353 2285 2285
query89 373 303 290 290
query90 1734 209 210 209
query91 143 143 111 111
query92 68 62 56 56
query93 1001 956 605 605
query94 706 314 196 196
query95 379 296 291 291
query96 490 567 277 277
query97 2683 2773 2608 2608
query98 224 211 208 208
query99 1345 1406 1271 1271
Total cold run time: 269202 ms
Total hot run time: 187397 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2644725745
<details>
<summary>TPC-DS: <b>Total hot run time: 183143 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8779c6557723172b587f0c4218e70793a3530092, data reload: false
query1 978 405 378 378
query2 6550 1874 1828 1828
query3 6806 213 217 213
query4 27057 23270 23434 23270
query5 4307 650 469 469
query6 293 218 191 191
query7 4609 509 317 317
query8 304 248 230 230
query9 8606 2489 2513 2489
query10 477 309 258 258
query11 15673 15095 14885 14885
query12 155 105 101 101
query13 1645 509 394 394
query14 9471 6090 6636 6090
query15 214 191 177 177
query16 7316 659 466 466
query17 1181 684 545 545
query18 1837 386 299 299
query19 193 193 162 162
query20 125 118 118 118
query21 210 122 100 100
query22 4175 4326 4337 4326
query23 33758 33197 32971 32971
query24 7724 2358 2386 2358
query25 585 480 422 422
query26 1229 274 162 162
query27 2170 520 348 348
query28 3957 2400 2364 2364
query29 741 535 400 400
query30 229 180 156 156
query31 970 843 793 793
query32 72 65 63 63
query33 558 353 296 296
query34 796 843 497 497
query35 797 797 720 720
query36 958 977 865 865
query37 120 97 79 79
query38 4105 4153 4105 4105
query39 1495 1410 1388 1388
query40 210 120 104 104
query41 73 53 54 53
query42 134 108 102 102
query43 498 518 462 462
query44 1303 789 793 789
query45 175 171 160 160
query46 859 1041 643 643
query47 1777 1803 1742 1742
query48 374 404 314 314
query49 794 492 432 432
query50 678 739 409 409
query51 4195 4126 4146 4126
query52 108 105 94 94
query53 230 254 188 188
query54 478 505 404 404
query55 80 80 79 79
query56 273 276 271 271
query57 1158 1158 1083 1083
query58 257 240 244 240
query59 2687 2601 2521 2521
query60 276 275 268 268
query61 119 117 122 117
query62 814 728 683 683
query63 222 193 184 184
query64 4383 1027 644 644
query65 3247 3091 3119 3091
query66 1119 431 302 302
query67 15886 15719 15609 15609
query68 2283 822 544 544
query69 423 307 267 267
query70 1187 1057 1069 1057
query71 326 316 265 265
query72 5889 3548 3769 3548
query73 645 744 350 350
query74 8965 9213 8708 8708
query75 3100 3148 2660 2660
query76 2313 1172 757 757
query77 340 363 281 281
query78 9847 10249 9346 9346
query79 1051 825 582 582
query80 1188 535 445 445
query81 517 274 239 239
query82 921 159 120 120
query83 268 179 158 158
query84 237 98 81 81
query85 1001 341 298 298
query86 377 312 279 279
query87 4361 4486 4368 4368
query88 2861 2247 2187 2187
query89 389 310 299 299
query90 1757 192 198 192
query91 140 139 110 110
query92 62 60 60 60
query93 1082 993 583 583
query94 655 416 284 284
query95 354 268 254 254
query96 489 556 268 268
query97 2785 2825 2692 2692
query98 220 206 204 204
query99 1304 1405 1250 1250
Total cold run time: 262190 ms
Total hot run time: 183143 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix incorrect result when there are insert after delete for same key in one load [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #41701:
URL: https://github.com/apache/doris/pull/41701#discussion_r1825398719
##########
be/src/olap/partial_update_info.cpp:
##########
@@ -658,4 +669,431 @@ Status FlexibleReadPlan::fill_non_primary_key_columns_for_row_store(
return Status::OK();
}
+BlockAggregator::BlockAggregator(segment_v2::VerticalSegmentWriter& vertical_segment_writer)
+ : _writer(vertical_segment_writer), _tablet_schema(*_writer._tablet_schema) {}
+
+void BlockAggregator::merge_one_row(vectorized::MutableBlock& dst_block,
+ vectorized::Block* src_block, int64_t rid,
+ BitmapValue& skip_bitmap) {
+ for (size_t cid {_tablet_schema.num_key_columns()}; cid < _tablet_schema.num_columns(); cid++) {
+ if (cid == _tablet_schema.skip_bitmap_col_idx()) {
+ auto& cur_skip_bitmap =
+ assert_cast<vectorized::ColumnBitmap*>(dst_block.mutable_columns()[cid].get())
+ ->get_data()
+ .back();
+ const auto& new_row_skip_bitmap =
+ assert_cast<vectorized::ColumnBitmap*>(
+ src_block->get_by_position(cid).column->assume_mutable().get())
+ ->get_data()[rid];
+ cur_skip_bitmap &= new_row_skip_bitmap;
+ continue;
+ }
+ if (!skip_bitmap.contains(_tablet_schema.column(cid).unique_id())) {
+ dst_block.mutable_columns()[cid]->pop_back(1);
+ dst_block.mutable_columns()[cid]->insert_from(*src_block->get_by_position(cid).column,
+ rid);
+ }
+ }
+ VLOG_DEBUG << fmt::format("merge a row, after merge, output_block.rows()={}, state: {}",
+ dst_block.rows(), _state.to_string());
+}
+
+void BlockAggregator::append_one_row(vectorized::MutableBlock& dst_block,
+ vectorized::Block* src_block, int64_t rid) {
+ dst_block.add_row(src_block, rid);
+ _state.rows++;
+ VLOG_DEBUG << fmt::format("append a new row, after append, output_block.rows()={}, state: {}",
+ dst_block.rows(), _state.to_string());
+}
+
+void BlockAggregator::remove_last_n_rows(vectorized::MutableBlock& dst_block, int n) {
+ if (n > 0) {
+ for (size_t cid {0}; cid < _tablet_schema.num_columns(); cid++) {
+ DCHECK_GE(dst_block.mutable_columns()[cid]->size(), n);
+ dst_block.mutable_columns()[cid]->pop_back(n);
+ }
+ }
+}
+
+void BlockAggregator::append_or_merge_row(vectorized::MutableBlock& dst_block,
+ vectorized::Block* src_block, int64_t rid,
+ BitmapValue& skip_bitmap, bool have_delete_sign) {
+ if (have_delete_sign) {
+ // remove all the previous batched rows
+ remove_last_n_rows(dst_block, _state.rows);
+ _state.rows = 0;
+ _state.has_row_with_delete_sign = true;
+
+ append_one_row(dst_block, src_block, rid);
+ } else {
+ if (_state.should_merge()) {
+ merge_one_row(dst_block, src_block, rid, skip_bitmap);
+ } else {
+ append_one_row(dst_block, src_block, rid);
Review Comment:
warning: function 'aggregate_rows' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
Status BlockAggregator::aggregate_rows(
^
```
<details>
<summary>Additional context</summary>
**be/src/olap/partial_update_info.cpp:731:** 84 lines including whitespace and comments (threshold 80)
```cpp
Status BlockAggregator::aggregate_rows(
^
```
</details>
##########
be/src/olap/partial_update_info.cpp:
##########
@@ -658,4 +669,431 @@
return Status::OK();
}
+BlockAggregator::BlockAggregator(segment_v2::VerticalSegmentWriter& vertical_segment_writer)
+ : _writer(vertical_segment_writer), _tablet_schema(*_writer._tablet_schema) {}
+
+void BlockAggregator::merge_one_row(vectorized::MutableBlock& dst_block,
+ vectorized::Block* src_block, int64_t rid,
+ BitmapValue& skip_bitmap) {
+ for (size_t cid {_tablet_schema.num_key_columns()}; cid < _tablet_schema.num_columns(); cid++) {
+ if (cid == _tablet_schema.skip_bitmap_col_idx()) {
+ auto& cur_skip_bitmap =
+ assert_cast<vectorized::ColumnBitmap*>(dst_block.mutable_columns()[cid].get())
+ ->get_data()
+ .back();
+ const auto& new_row_skip_bitmap =
+ assert_cast<vectorized::ColumnBitmap*>(
+ src_block->get_by_position(cid).column->assume_mutable().get())
+ ->get_data()[rid];
+ cur_skip_bitmap &= new_row_skip_bitmap;
+ continue;
+ }
+ if (!skip_bitmap.contains(_tablet_schema.column(cid).unique_id())) {
+ dst_block.mutable_columns()[cid]->pop_back(1);
+ dst_block.mutable_columns()[cid]->insert_from(*src_block->get_by_position(cid).column,
+ rid);
+ }
+ }
+ VLOG_DEBUG << fmt::format("merge a row, after merge, output_block.rows()={}, state: {}",
+ dst_block.rows(), _state.to_string());
+}
+
+void BlockAggregator::append_one_row(vectorized::MutableBlock& dst_block,
+ vectorized::Block* src_block, int64_t rid) {
+ dst_block.add_row(src_block, rid);
+ _state.rows++;
+ VLOG_DEBUG << fmt::format("append a new row, after append, output_block.rows()={}, state: {}",
+ dst_block.rows(), _state.to_string());
+}
+
+void BlockAggregator::remove_last_n_rows(vectorized::MutableBlock& dst_block, int n) {
+ if (n > 0) {
+ for (size_t cid {0}; cid < _tablet_schema.num_columns(); cid++) {
+ DCHECK_GE(dst_block.mutable_columns()[cid]->size(), n);
+ dst_block.mutable_columns()[cid]->pop_back(n);
+ }
+ }
+}
+
+void BlockAggregator::append_or_merge_row(vectorized::MutableBlock& dst_block,
+ vectorized::Block* src_block, int64_t rid,
+ BitmapValue& skip_bitmap, bool have_delete_sign) {
+ if (have_delete_sign) {
+ // remove all the previous batched rows
+ remove_last_n_rows(dst_block, _state.rows);
+ _state.rows = 0;
+ _state.has_row_with_delete_sign = true;
+
+ append_one_row(dst_block, src_block, rid);
+ } else {
+ if (_state.should_merge()) {
+ merge_one_row(dst_block, src_block, rid, skip_bitmap);
+ } else {
+ append_one_row(dst_block, src_block, rid);
+ }
+ }
+};
+
+Status BlockAggregator::aggregate_rows(
+ vectorized::MutableBlock& output_block, vectorized::Block* block, int64_t start,
+ int64_t end, std::string key, std::vector<BitmapValue>* skip_bitmaps,
+ const signed char* delete_signs, vectorized::IOlapColumnDataAccessor* seq_column,
+ const std::vector<RowsetSharedPtr>& specified_rowsets,
+ std::vector<std::unique_ptr<SegmentCacheHandle>>& segment_caches) {
+ VLOG_DEBUG << fmt::format("merge rows in range=[{}-{})", start, end);
+ if (end - start == 1) {
+ output_block.add_row(block, start);
+ VLOG_DEBUG << fmt::format("append a row directly, rid={}", start);
+ return Status::OK();
+ }
+
+ auto seq_col_unique_id = _tablet_schema.column(_tablet_schema.sequence_col_idx()).unique_id();
+ auto delete_sign_col_unique_id =
+ _tablet_schema.column(_tablet_schema.delete_sign_idx()).unique_id();
+
+ _state.reset();
+
+ RowLocation loc;
+ RowsetSharedPtr rowset;
+ std::string previous_encoded_seq_value {};
+ Status st = _writer._tablet->lookup_row_key(
+ key, &_tablet_schema, false, specified_rowsets, &loc, _writer._mow_context->max_version,
+ segment_caches, &rowset, true, &previous_encoded_seq_value);
+ int64_t pos = start;
+ bool is_expected_st = (st.is<ErrorCode::KEY_NOT_FOUND>() || st.ok());
+ DCHECK(is_expected_st || st.is<ErrorCode::MEM_LIMIT_EXCEEDED>())
+ << "[BlockAggregator::aggregate_rows] unexpected error status while lookup_row_key:"
+ << st;
+ if (!is_expected_st) {
+ return st;
+ }
+
+ std::string cur_seq_val;
+ if (st.ok()) {
+ for (pos = start; pos < end; pos++) {
+ auto& skip_bitmap = skip_bitmaps->at(pos);
+ bool row_has_sequence_col = (!skip_bitmap.contains(seq_col_unique_id));
+ // Discard all the rows whose seq value is smaller than previous_encoded_seq_value.
+ if (row_has_sequence_col) {
+ std::string seq_val {};
+ _writer._encode_seq_column(seq_column, pos, &seq_val);
+ if (Slice {seq_val}.compare(Slice {previous_encoded_seq_value}) < 0) {
+ continue;
+ }
+ cur_seq_val = std::move(seq_val);
+ break;
+ }
+ cur_seq_val = std::move(previous_encoded_seq_value);
+ break;
+ }
+ } else {
+ pos = start;
+ auto& skip_bitmap = skip_bitmaps->at(pos);
+ bool row_has_sequence_col = (!skip_bitmap.contains(seq_col_unique_id));
+ if (row_has_sequence_col) {
+ std::string seq_val {};
+ // for rows that don't specify seqeunce col, seq_val will be encoded to minial value
+ _writer._encode_seq_column(seq_column, pos, &seq_val);
+ cur_seq_val = std::move(seq_val);
+ } else {
+ cur_seq_val.clear();
+ RETURN_IF_ERROR(_writer._generate_encoded_default_seq_value(
+ _tablet_schema, *_writer._opts.rowset_ctx->partial_update_info, &cur_seq_val));
+ }
+ }
+ VLOG_DEBUG << fmt::format("start pos={}, output_block.rows()={}", pos, output_block.rows());
+
+ for (int64_t rid {pos}; rid < end; rid++) {
+ auto& skip_bitmap = skip_bitmaps->at(rid);
+ bool row_has_sequence_col = (!skip_bitmap.contains(seq_col_unique_id));
+ bool have_delete_sign =
+ (!skip_bitmap.contains(delete_sign_col_unique_id) && delete_signs[rid] != 0);
+ if (!row_has_sequence_col) {
+ append_or_merge_row(output_block, block, rid, skip_bitmap, have_delete_sign);
+ } else {
+ std::string seq_val {};
+ _writer._encode_seq_column(seq_column, rid, &seq_val);
+ if (Slice {seq_val}.compare(Slice {cur_seq_val}) >= 0) {
+ append_or_merge_row(output_block, block, rid, skip_bitmap, have_delete_sign);
+ cur_seq_val = std::move(seq_val);
+ } else {
+ VLOG_DEBUG << fmt::format(
+ "skip rid={} becasue its seq value is lower than the previous", rid);
+ }
+ }
+ }
+ return Status::OK();
+};
+
+Status BlockAggregator::aggregate_for_sequence_column(
+ vectorized::Block* block, size_t num_rows,
+ const std::vector<vectorized::IOlapColumnDataAccessor*>& key_columns,
+ vectorized::IOlapColumnDataAccessor* seq_column,
+ const std::vector<RowsetSharedPtr>& specified_rowsets,
+ std::vector<std::unique_ptr<SegmentCacheHandle>>& segment_caches) {
+ DCHECK_EQ(block->columns(), _tablet_schema.num_columns());
+ VLOG_DEBUG << fmt::format("BlockAggregator::aggregate_for_sequence_column enter: block:{}\n",
+ block->dump_data());
+ // the process logic here is the same as MemTable::_aggregate_for_flexible_partial_update_without_seq_col()
+ // after this function, there will be at most 2 rows for a specified key
+ std::vector<BitmapValue>* skip_bitmaps =
+ &(assert_cast<vectorized::ColumnBitmap*>(
+ block->get_by_position(_tablet_schema.skip_bitmap_col_idx())
+ .column->assume_mutable()
+ .get())
+ ->get_data());
+ const auto* delete_signs = BaseTablet::get_delete_sign_column_data(*block, num_rows);
+
+ auto filtered_block = _tablet_schema.create_block();
+ vectorized::MutableBlock output_block =
+ vectorized::MutableBlock::build_mutable_block(&filtered_block);
+
+ int same_key_rows {0};
+ std::string previous_key {};
+ for (size_t block_pos {0}; block_pos < num_rows; block_pos++) {
+ std::string key = _writer._full_encode_keys(key_columns, block_pos);
+ if (block_pos > 0 && previous_key == key) {
+ same_key_rows++;
+ } else {
+ if (same_key_rows > 0) {
+ RETURN_IF_ERROR(aggregate_rows(output_block, block, block_pos - same_key_rows,
+ block_pos, std::move(previous_key), skip_bitmaps,
+ delete_signs, seq_column, specified_rowsets,
+ segment_caches));
+ }
+ same_key_rows = 1;
+ }
+ previous_key = std::move(key);
+ }
+ if (same_key_rows > 0) {
+ RETURN_IF_ERROR(aggregate_rows(output_block, block, num_rows - same_key_rows, num_rows,
+ std::move(previous_key), skip_bitmaps, delete_signs,
+ seq_column, specified_rowsets, segment_caches));
+ }
+
+ block->swap(output_block.to_block());
+ VLOG_DEBUG << fmt::format(
+ "BlockAggregator::aggregate_for_sequence_column after: data.block:{}\n",
+ block->dump_data());
+ return Status::OK();
+}
+
+Status BlockAggregator::fill_sequence_column(vectorized::Block* block, size_t num_rows,
+ const FixedReadPlan& read_plan,
+ std::vector<BitmapValue>& skip_bitmaps) {
+ DCHECK(_tablet_schema.has_sequence_col());
+ std::vector<uint32_t> cids {static_cast<uint32_t>(_tablet_schema.sequence_col_idx())};
+ auto seq_col_unique_id = _tablet_schema.column(_tablet_schema.sequence_col_idx()).unique_id();
+
+ auto seq_col_block = _tablet_schema.create_block_by_cids(cids);
+ auto tmp_block = _tablet_schema.create_block_by_cids(cids);
+ std::map<uint32_t, uint32_t> read_index;
+ RETURN_IF_ERROR(read_plan.read_columns_by_plan(_tablet_schema, cids, _writer._rsid_to_rowset,
+ seq_col_block, &read_index));
+
+ auto new_seq_col_ptr = tmp_block.get_by_position(0).column->assume_mutable();
+ const auto& old_seq_col_ptr = *seq_col_block.get_by_position(0).column;
+ const auto& cur_seq_col_ptr = *block->get_by_position(_tablet_schema.sequence_col_idx()).column;
+ for (size_t block_pos {0}; block_pos < num_rows; block_pos++) {
+ if (read_index.contains(block_pos)) {
+ new_seq_col_ptr->insert_from(old_seq_col_ptr, read_index[block_pos]);
+ skip_bitmaps[block_pos].remove(seq_col_unique_id);
+ } else {
+ new_seq_col_ptr->insert_from(cur_seq_col_ptr, block_pos);
+ }
+ }
Review Comment:
warning: function 'aggregate_for_insert_after_delete' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
Status BlockAggregator::aggregate_for_insert_after_delete(
^
```
<details>
<summary>Additional context</summary>
**be/src/olap/partial_update_info.cpp:903:** 96 lines including whitespace and comments (threshold 80)
```cpp
Status BlockAggregator::aggregate_for_insert_after_delete(
^
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix incorrect result when there are insert after delete for same key in one load [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2443461550
<details>
<summary>ClickBench: <b>Total hot run time: 33.37 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2bde941d7f0c31261a07d9bdc7873b8f1521d792, data reload: false
query1 0.04 0.03 0.03
query2 0.07 0.02 0.03
query3 0.23 0.07 0.06
query4 1.64 0.10 0.11
query5 0.39 0.40 0.40
query6 1.14 0.65 0.65
query7 0.02 0.02 0.01
query8 0.04 0.05 0.04
query9 0.55 0.49 0.50
query10 0.55 0.53 0.53
query11 0.14 0.11 0.11
query12 0.13 0.12 0.11
query13 0.61 0.60 0.59
query14 2.72 2.82 2.80
query15 0.91 0.82 0.83
query16 0.41 0.38 0.39
query17 1.01 0.99 1.00
query18 0.24 0.22 0.22
query19 1.88 1.88 1.90
query20 0.01 0.01 0.02
query21 15.35 0.60 0.55
query22 2.90 2.24 2.72
query23 17.05 0.91 0.80
query24 2.55 1.40 2.28
query25 0.19 0.08 0.16
query26 0.55 0.12 0.13
query27 0.07 0.04 0.05
query28 9.81 1.12 1.07
query29 12.59 3.27 3.23
query30 0.24 0.06 0.06
query31 2.88 0.38 0.38
query32 3.27 0.46 0.46
query33 2.98 3.06 3.00
query34 17.08 4.48 4.49
query35 4.56 4.64 4.44
query36 0.67 0.50 0.49
query37 0.08 0.06 0.06
query38 0.04 0.03 0.04
query39 0.03 0.02 0.03
query40 0.16 0.12 0.12
query41 0.07 0.02 0.02
query42 0.03 0.02 0.02
query43 0.04 0.03 0.02
Total cold run time: 105.92 s
Total hot run time: 33.37 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #41701:
URL: https://github.com/apache/doris/pull/41701#discussion_r1833947591
##########
be/src/olap/partial_update_info.cpp:
##########
@@ -536,22 +569,41 @@
const vectorized::IColumn& default_value_col,
const vectorized::IColumn& old_value_col,
const vectorized::IColumn& cur_col, std::size_t block_pos,
- std::size_t segment_pos, bool skipped, bool use_default,
- const signed char* delete_sign_column_data) {
+ std::size_t segment_pos, bool skipped, bool row_has_sequence_col,
+ bool use_default, const signed char* delete_sign_column_data) {
if (skipped) {
DCHECK(cid != tablet_schema.skip_bitmap_col_idx());
DCHECK(cid != tablet_schema.version_col_idx());
DCHECK(!tablet_column.is_row_store_column());
- auto delete_sign_pos = read_index[tablet_schema.delete_sign_idx()][segment_pos];
- if (use_default || (delete_sign_column_data != nullptr &&
- delete_sign_column_data[delete_sign_pos] != 0)) {
+ if (!use_default) {
+ if (delete_sign_column_data != nullptr) {
+ auto it = read_index[tablet_schema.delete_sign_idx()].find(segment_pos);
+ if (it != read_index[tablet_schema.delete_sign_idx()].end()) {
+ if ((delete_sign_column_data[it->second] != 0) &&
+ !(tablet_schema.has_sequence_col() && !row_has_sequence_col)) {
Review Comment:
warning: boolean expression can be simplified by DeMorgan's theorem [readability-simplify-boolean-expr]
```suggestion
(!tablet_schema.has_sequence_col() || row_has_sequence_col)) {
```
##########
be/src/olap/partial_update_info.cpp:
##########
@@ -353,8 +367,19 @@ Status FixedReadPlan::fill_missing_columns(
// to check if a row REALLY exists in the table.
auto segment_pos = idx + segment_start_pos;
auto pos_in_old_block = read_index[segment_pos];
- if (use_default_or_null_flag[idx] || (delete_sign_column_data != nullptr &&
- delete_sign_column_data[pos_in_old_block] != 0)) {
+
+ bool should_use_default = use_default_or_null_flag[idx];
+ if (!should_use_default) {
+ if ((delete_sign_column_data != nullptr &&
+ delete_sign_column_data[pos_in_old_block] != 0) &&
+ !(tablet_schema.has_sequence_col() && !have_input_seq_column)) {
Review Comment:
warning: boolean expression can be simplified by DeMorgan's theorem [readability-simplify-boolean-expr]
```suggestion
(!tablet_schema.has_sequence_col() || have_input_seq_column)) {
```
##########
be/src/olap/partial_update_info.cpp:
##########
@@ -610,22 +669,40 @@
const vectorized::IColumn& default_value_col,
const vectorized::IColumn& old_value_col,
const vectorized::IColumn& cur_col, std::size_t block_pos,
- bool skipped, bool use_default,
+ bool skipped, bool row_has_sequence_col, bool use_default,
const signed char* delete_sign_column_data,
uint32_t pos_in_old_block) {
if (skipped) {
DCHECK(cid != tablet_schema.skip_bitmap_col_idx());
DCHECK(cid != tablet_schema.version_col_idx());
DCHECK(!tablet_column.is_row_store_column());
-
- if (use_default || (delete_sign_column_data != nullptr &&
- delete_sign_column_data[pos_in_old_block] != 0)) {
+ if (!use_default) {
+ if ((delete_sign_column_data != nullptr &&
+ delete_sign_column_data[pos_in_old_block] != 0) &&
+ !(tablet_schema.has_sequence_col() && !row_has_sequence_col)) {
Review Comment:
warning: boolean expression can be simplified by DeMorgan's theorem [readability-simplify-boolean-expr]
```suggestion
(!tablet_schema.has_sequence_col() || row_has_sequence_col)) {
```
##########
be/src/olap/partial_update_info.cpp:
##########
@@ -652,10 +729,440 @@
*old_value_block.get_by_position(i).column,
*block->get_by_position(cid).column, block_pos,
skip_bitmaps->at(block_pos).contains(col_uid),
+ tablet_schema.has_sequence_col()
+ ? !skip_bitmaps->at(block_pos).contains(seq_col_unique_id)
+ : false,
use_default_or_null_flag[idx], delete_sign_column_data, pos_in_old_block);
}
}
return Status::OK();
}
+BlockAggregator::BlockAggregator(segment_v2::VerticalSegmentWriter& vertical_segment_writer)
+ : _writer(vertical_segment_writer), _tablet_schema(*_writer._tablet_schema) {}
+
+void BlockAggregator::merge_one_row(vectorized::MutableBlock& dst_block,
+ vectorized::Block* src_block, int64_t rid,
+ BitmapValue& skip_bitmap) {
+ for (size_t cid {_tablet_schema.num_key_columns()}; cid < _tablet_schema.num_columns(); cid++) {
+ if (cid == _tablet_schema.skip_bitmap_col_idx()) {
+ auto& cur_skip_bitmap =
+ assert_cast<vectorized::ColumnBitmap*>(dst_block.mutable_columns()[cid].get())
+ ->get_data()
+ .back();
+ const auto& new_row_skip_bitmap =
+ assert_cast<vectorized::ColumnBitmap*>(
+ src_block->get_by_position(cid).column->assume_mutable().get())
+ ->get_data()[rid];
+ cur_skip_bitmap &= new_row_skip_bitmap;
+ continue;
+ }
+ if (!skip_bitmap.contains(_tablet_schema.column(cid).unique_id())) {
+ dst_block.mutable_columns()[cid]->pop_back(1);
+ dst_block.mutable_columns()[cid]->insert_from(*src_block->get_by_position(cid).column,
+ rid);
+ }
+ }
+ VLOG_DEBUG << fmt::format("merge a row, after merge, output_block.rows()={}, state: {}",
+ dst_block.rows(), _state.to_string());
+}
+
+void BlockAggregator::append_one_row(vectorized::MutableBlock& dst_block,
+ vectorized::Block* src_block, int64_t rid) {
+ dst_block.add_row(src_block, rid);
+ _state.rows++;
+ VLOG_DEBUG << fmt::format("append a new row, after append, output_block.rows()={}, state: {}",
+ dst_block.rows(), _state.to_string());
+}
+
+void BlockAggregator::remove_last_n_rows(vectorized::MutableBlock& dst_block, int n) {
+ if (n > 0) {
+ for (size_t cid {0}; cid < _tablet_schema.num_columns(); cid++) {
+ DCHECK_GE(dst_block.mutable_columns()[cid]->size(), n);
+ dst_block.mutable_columns()[cid]->pop_back(n);
+ }
+ }
+}
+
+void BlockAggregator::append_or_merge_row(vectorized::MutableBlock& dst_block,
+ vectorized::Block* src_block, int64_t rid,
+ BitmapValue& skip_bitmap, bool have_delete_sign) {
+ if (have_delete_sign) {
+ // remove all the previous batched rows
+ remove_last_n_rows(dst_block, _state.rows);
+ _state.rows = 0;
+ _state.has_row_with_delete_sign = true;
+
+ append_one_row(dst_block, src_block, rid);
+ } else {
+ if (_state.should_merge()) {
+ merge_one_row(dst_block, src_block, rid, skip_bitmap);
+ } else {
+ append_one_row(dst_block, src_block, rid);
+ }
+ }
+};
+
+Status BlockAggregator::aggregate_rows(
Review Comment:
warning: function 'aggregate_rows' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
Status BlockAggregator::aggregate_rows(
^
```
<details>
<summary>Additional context</summary>
**be/src/olap/partial_update_info.cpp:805:** 84 lines including whitespace and comments (threshold 80)
```cpp
Status BlockAggregator::aggregate_rows(
^
```
</details>
##########
be/src/olap/partial_update_info.cpp:
##########
@@ -652,10 +729,440 @@
*old_value_block.get_by_position(i).column,
*block->get_by_position(cid).column, block_pos,
skip_bitmaps->at(block_pos).contains(col_uid),
+ tablet_schema.has_sequence_col()
+ ? !skip_bitmaps->at(block_pos).contains(seq_col_unique_id)
+ : false,
use_default_or_null_flag[idx], delete_sign_column_data, pos_in_old_block);
}
}
return Status::OK();
}
+BlockAggregator::BlockAggregator(segment_v2::VerticalSegmentWriter& vertical_segment_writer)
+ : _writer(vertical_segment_writer), _tablet_schema(*_writer._tablet_schema) {}
+
+void BlockAggregator::merge_one_row(vectorized::MutableBlock& dst_block,
+ vectorized::Block* src_block, int64_t rid,
+ BitmapValue& skip_bitmap) {
+ for (size_t cid {_tablet_schema.num_key_columns()}; cid < _tablet_schema.num_columns(); cid++) {
+ if (cid == _tablet_schema.skip_bitmap_col_idx()) {
+ auto& cur_skip_bitmap =
+ assert_cast<vectorized::ColumnBitmap*>(dst_block.mutable_columns()[cid].get())
+ ->get_data()
+ .back();
+ const auto& new_row_skip_bitmap =
+ assert_cast<vectorized::ColumnBitmap*>(
+ src_block->get_by_position(cid).column->assume_mutable().get())
+ ->get_data()[rid];
+ cur_skip_bitmap &= new_row_skip_bitmap;
+ continue;
+ }
+ if (!skip_bitmap.contains(_tablet_schema.column(cid).unique_id())) {
+ dst_block.mutable_columns()[cid]->pop_back(1);
+ dst_block.mutable_columns()[cid]->insert_from(*src_block->get_by_position(cid).column,
+ rid);
+ }
+ }
+ VLOG_DEBUG << fmt::format("merge a row, after merge, output_block.rows()={}, state: {}",
+ dst_block.rows(), _state.to_string());
+}
+
+void BlockAggregator::append_one_row(vectorized::MutableBlock& dst_block,
+ vectorized::Block* src_block, int64_t rid) {
+ dst_block.add_row(src_block, rid);
+ _state.rows++;
+ VLOG_DEBUG << fmt::format("append a new row, after append, output_block.rows()={}, state: {}",
+ dst_block.rows(), _state.to_string());
+}
+
+void BlockAggregator::remove_last_n_rows(vectorized::MutableBlock& dst_block, int n) {
+ if (n > 0) {
+ for (size_t cid {0}; cid < _tablet_schema.num_columns(); cid++) {
+ DCHECK_GE(dst_block.mutable_columns()[cid]->size(), n);
+ dst_block.mutable_columns()[cid]->pop_back(n);
+ }
+ }
+}
+
+void BlockAggregator::append_or_merge_row(vectorized::MutableBlock& dst_block,
+ vectorized::Block* src_block, int64_t rid,
+ BitmapValue& skip_bitmap, bool have_delete_sign) {
+ if (have_delete_sign) {
+ // remove all the previous batched rows
+ remove_last_n_rows(dst_block, _state.rows);
+ _state.rows = 0;
+ _state.has_row_with_delete_sign = true;
+
+ append_one_row(dst_block, src_block, rid);
+ } else {
+ if (_state.should_merge()) {
+ merge_one_row(dst_block, src_block, rid, skip_bitmap);
+ } else {
+ append_one_row(dst_block, src_block, rid);
+ }
+ }
+};
+
+Status BlockAggregator::aggregate_rows(
+ vectorized::MutableBlock& output_block, vectorized::Block* block, int64_t start,
+ int64_t end, std::string key, std::vector<BitmapValue>* skip_bitmaps,
+ const signed char* delete_signs, vectorized::IOlapColumnDataAccessor* seq_column,
+ const std::vector<RowsetSharedPtr>& specified_rowsets,
+ std::vector<std::unique_ptr<SegmentCacheHandle>>& segment_caches) {
+ VLOG_DEBUG << fmt::format("merge rows in range=[{}-{})", start, end);
+ if (end - start == 1) {
+ output_block.add_row(block, start);
+ VLOG_DEBUG << fmt::format("append a row directly, rid={}", start);
+ return Status::OK();
+ }
+
+ auto seq_col_unique_id = _tablet_schema.column(_tablet_schema.sequence_col_idx()).unique_id();
+ auto delete_sign_col_unique_id =
+ _tablet_schema.column(_tablet_schema.delete_sign_idx()).unique_id();
+
+ _state.reset();
+
+ RowLocation loc;
+ RowsetSharedPtr rowset;
+ std::string previous_encoded_seq_value {};
+ Status st = _writer._tablet->lookup_row_key(
+ key, &_tablet_schema, false, specified_rowsets, &loc, _writer._mow_context->max_version,
+ segment_caches, &rowset, true, &previous_encoded_seq_value);
+ int64_t pos = start;
+ bool is_expected_st = (st.is<ErrorCode::KEY_NOT_FOUND>() || st.ok());
+ DCHECK(is_expected_st || st.is<ErrorCode::MEM_LIMIT_EXCEEDED>())
+ << "[BlockAggregator::aggregate_rows] unexpected error status while lookup_row_key:"
+ << st;
+ if (!is_expected_st) {
+ return st;
+ }
+
+ std::string cur_seq_val;
+ if (st.ok()) {
+ for (pos = start; pos < end; pos++) {
+ auto& skip_bitmap = skip_bitmaps->at(pos);
+ bool row_has_sequence_col = (!skip_bitmap.contains(seq_col_unique_id));
+ // Discard all the rows whose seq value is smaller than previous_encoded_seq_value.
+ if (row_has_sequence_col) {
+ std::string seq_val {};
+ _writer._encode_seq_column(seq_column, pos, &seq_val);
+ if (Slice {seq_val}.compare(Slice {previous_encoded_seq_value}) < 0) {
+ continue;
+ }
+ cur_seq_val = std::move(seq_val);
+ break;
+ }
+ cur_seq_val = std::move(previous_encoded_seq_value);
+ break;
+ }
+ } else {
+ pos = start;
+ auto& skip_bitmap = skip_bitmaps->at(pos);
+ bool row_has_sequence_col = (!skip_bitmap.contains(seq_col_unique_id));
+ if (row_has_sequence_col) {
+ std::string seq_val {};
+ // for rows that don't specify seqeunce col, seq_val will be encoded to minial value
+ _writer._encode_seq_column(seq_column, pos, &seq_val);
+ cur_seq_val = std::move(seq_val);
+ } else {
+ cur_seq_val.clear();
+ RETURN_IF_ERROR(_writer._generate_encoded_default_seq_value(
+ _tablet_schema, *_writer._opts.rowset_ctx->partial_update_info, &cur_seq_val));
+ }
+ }
+ VLOG_DEBUG << fmt::format("start pos={}, output_block.rows()={}", pos, output_block.rows());
+
+ for (int64_t rid {pos}; rid < end; rid++) {
+ auto& skip_bitmap = skip_bitmaps->at(rid);
+ bool row_has_sequence_col = (!skip_bitmap.contains(seq_col_unique_id));
+ bool have_delete_sign =
+ (!skip_bitmap.contains(delete_sign_col_unique_id) && delete_signs[rid] != 0);
+ if (!row_has_sequence_col) {
+ append_or_merge_row(output_block, block, rid, skip_bitmap, have_delete_sign);
+ } else {
+ std::string seq_val {};
+ _writer._encode_seq_column(seq_column, rid, &seq_val);
+ if (Slice {seq_val}.compare(Slice {cur_seq_val}) >= 0) {
+ append_or_merge_row(output_block, block, rid, skip_bitmap, have_delete_sign);
+ cur_seq_val = std::move(seq_val);
+ } else {
+ VLOG_DEBUG << fmt::format(
+ "skip rid={} becasue its seq value is lower than the previous", rid);
+ }
+ }
+ }
+ return Status::OK();
+};
+
+Status BlockAggregator::aggregate_for_sequence_column(
+ vectorized::Block* block, size_t num_rows,
+ const std::vector<vectorized::IOlapColumnDataAccessor*>& key_columns,
+ vectorized::IOlapColumnDataAccessor* seq_column,
+ const std::vector<RowsetSharedPtr>& specified_rowsets,
+ std::vector<std::unique_ptr<SegmentCacheHandle>>& segment_caches) {
+ DCHECK_EQ(block->columns(), _tablet_schema.num_columns());
+ VLOG_DEBUG << fmt::format("BlockAggregator::aggregate_for_sequence_column enter: block:{}\n",
+ block->dump_data());
+ // the process logic here is the same as MemTable::_aggregate_for_flexible_partial_update_without_seq_col()
+ // after this function, there will be at most 2 rows for a specified key
+ std::vector<BitmapValue>* skip_bitmaps =
+ &(assert_cast<vectorized::ColumnBitmap*>(
+ block->get_by_position(_tablet_schema.skip_bitmap_col_idx())
+ .column->assume_mutable()
+ .get())
+ ->get_data());
+ const auto* delete_signs = BaseTablet::get_delete_sign_column_data(*block, num_rows);
+
+ auto filtered_block = _tablet_schema.create_block();
+ vectorized::MutableBlock output_block =
+ vectorized::MutableBlock::build_mutable_block(&filtered_block);
+
+ int same_key_rows {0};
+ std::string previous_key {};
+ for (size_t block_pos {0}; block_pos < num_rows; block_pos++) {
+ std::string key = _writer._full_encode_keys(key_columns, block_pos);
+ if (block_pos > 0 && previous_key == key) {
+ same_key_rows++;
+ } else {
+ if (same_key_rows > 0) {
+ RETURN_IF_ERROR(aggregate_rows(output_block, block, block_pos - same_key_rows,
+ block_pos, std::move(previous_key), skip_bitmaps,
+ delete_signs, seq_column, specified_rowsets,
+ segment_caches));
+ }
+ same_key_rows = 1;
+ }
+ previous_key = std::move(key);
+ }
+ if (same_key_rows > 0) {
+ RETURN_IF_ERROR(aggregate_rows(output_block, block, num_rows - same_key_rows, num_rows,
+ std::move(previous_key), skip_bitmaps, delete_signs,
+ seq_column, specified_rowsets, segment_caches));
+ }
+
+ block->swap(output_block.to_block());
+ VLOG_DEBUG << fmt::format(
+ "BlockAggregator::aggregate_for_sequence_column after: data.block:{}\n",
+ block->dump_data());
+ return Status::OK();
+}
+
+Status BlockAggregator::fill_sequence_column(vectorized::Block* block, size_t num_rows,
+ const FixedReadPlan& read_plan,
+ std::vector<BitmapValue>& skip_bitmaps) {
+ DCHECK(_tablet_schema.has_sequence_col());
+ std::vector<uint32_t> cids {static_cast<uint32_t>(_tablet_schema.sequence_col_idx())};
+ auto seq_col_unique_id = _tablet_schema.column(_tablet_schema.sequence_col_idx()).unique_id();
+
+ auto seq_col_block = _tablet_schema.create_block_by_cids(cids);
+ auto tmp_block = _tablet_schema.create_block_by_cids(cids);
+ std::map<uint32_t, uint32_t> read_index;
+ RETURN_IF_ERROR(read_plan.read_columns_by_plan(_tablet_schema, cids, _writer._rsid_to_rowset,
+ seq_col_block, &read_index));
+
+ auto new_seq_col_ptr = tmp_block.get_by_position(0).column->assume_mutable();
+ const auto& old_seq_col_ptr = *seq_col_block.get_by_position(0).column;
+ const auto& cur_seq_col_ptr = *block->get_by_position(_tablet_schema.sequence_col_idx()).column;
+ for (size_t block_pos {0}; block_pos < num_rows; block_pos++) {
+ if (read_index.contains(block_pos)) {
+ new_seq_col_ptr->insert_from(old_seq_col_ptr, read_index[block_pos]);
+ skip_bitmaps[block_pos].remove(seq_col_unique_id);
+ } else {
+ new_seq_col_ptr->insert_from(cur_seq_col_ptr, block_pos);
+ }
+ }
+ block->replace_by_position(_tablet_schema.sequence_col_idx(), std::move(new_seq_col_ptr));
+ return Status::OK();
+}
+
+Status BlockAggregator::aggregate_for_insert_after_delete(
Review Comment:
warning: function 'aggregate_for_insert_after_delete' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
Status BlockAggregator::aggregate_for_insert_after_delete(
^
```
<details>
<summary>Additional context</summary>
**be/src/olap/partial_update_info.cpp:977:** 96 lines including whitespace and comments (threshold 80)
```cpp
Status BlockAggregator::aggregate_for_insert_after_delete(
^
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix incorrect result when there are insert after delete for same key in one load [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #41701:
URL: https://github.com/apache/doris/pull/41701#discussion_r1797686781
##########
be/src/olap/memtable.cpp:
##########
@@ -578,6 +539,208 @@
}
}
+template <bool is_final>
+void MemTable::_aggregate_for_flexible_partial_update_without_seq_col(
+ const vectorized::ColumnsWithTypeAndName& block_data,
+ vectorized::MutableBlock& mutable_block, std::vector<RowInBlock*>& temp_row_in_blocks) {
+ RowInBlock* prev_row = nullptr;
+ int row_pos = -1;
+ auto& skip_bitmaps = assert_cast<vectorized::ColumnBitmap*>(
+ mutable_block.mutable_columns()[_skip_bitmap_col_idx].get())
+ ->get_data();
+ auto& delete_signs = assert_cast<vectorized::ColumnInt8*>(
+ mutable_block.mutable_columns()[_delete_sign_col_idx].get())
+ ->get_data();
+ RowInBlock* row_with_delete_sign {nullptr};
+ RowInBlock* row_without_delete_sign {nullptr};
+
+ auto finalize_rows = [&]() {
+ if (row_with_delete_sign != nullptr) {
+ temp_row_in_blocks.push_back(row_with_delete_sign);
+ _finalize_one_row<is_final, false>(row_with_delete_sign, block_data, ++row_pos);
+ row_with_delete_sign = nullptr;
+ }
+ if (row_without_delete_sign != nullptr) {
+ temp_row_in_blocks.push_back(row_without_delete_sign);
+ _finalize_one_row<is_final, false>(row_without_delete_sign, block_data, ++row_pos);
+ row_without_delete_sign = nullptr;
+ }
+ _arena->clear();
+ };
+
+ auto add_row = [&](RowInBlock* row, bool with_delete_sign) {
+ if (with_delete_sign) {
+ row_with_delete_sign = row;
+ } else {
+ row_without_delete_sign = row;
+ }
+ };
+ for (RowInBlock* cur_row : _row_in_blocks) {
+ const BitmapValue& skip_bitmap = skip_bitmaps[cur_row->_row_pos];
+ bool cur_row_has_delete_sign = (!skip_bitmap.contains(_delete_sign_col_unique_id) &&
+ delete_signs[cur_row->_row_pos] != 0);
+ prev_row =
+ (row_with_delete_sign == nullptr) ? row_without_delete_sign : row_with_delete_sign;
+ // compare keys, the keys of row_with_delete_sign and row_without_delete_sign is the same,
+ // choose any of them if it's valid
+ if (prev_row != nullptr && (*_vec_row_comparator)(prev_row, cur_row) == 0) {
+ if (cur_row_has_delete_sign) {
+ if (row_without_delete_sign != nullptr) {
+ // if there exits row without delete sign, remove it first
+ _clear_row_agg(row_without_delete_sign);
+ _stat.merged_rows++;
+ row_without_delete_sign = nullptr;
+ }
+ // and then unconditionally replace the previous row
+ prev_row = row_with_delete_sign;
+ } else {
+ prev_row = row_without_delete_sign;
+ }
+
+ if (prev_row == nullptr) {
+ add_row(cur_row, cur_row_has_delete_sign);
+ } else {
+ if (!prev_row->has_init_agg()) {
+ _init_row_for_agg(prev_row, mutable_block);
+ }
+ _stat.merged_rows++;
+ _aggregate_two_row_in_block<true>(mutable_block, cur_row, prev_row);
+ }
+ } else {
+ finalize_rows();
+ add_row(cur_row, cur_row_has_delete_sign);
+ }
+ }
+ // finalize the last lows
+ finalize_rows();
+}
+
+template <bool is_final>
+void MemTable::_aggregate_for_flexible_partial_update_with_seq_col(
Review Comment:
warning: function '_aggregate_for_flexible_partial_update_with_seq_col' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
void MemTable::_aggregate_for_flexible_partial_update_with_seq_col(
^
```
<details>
<summary>Additional context</summary>
**be/src/olap/memtable.cpp:618:** 121 lines including whitespace and comments (threshold 80)
```cpp
void MemTable::_aggregate_for_flexible_partial_update_with_seq_col(
^
```
</details>
##########
be/src/olap/memtable.cpp:
##########
@@ -452,6 +456,28 @@ void MemTable::_finalize_one_row(RowInBlock* row,
}
}
+void MemTable::_init_row_for_agg(RowInBlock* row, vectorized::MutableBlock& mutable_block) {
+ row->init_agg_places(_arena->aligned_alloc(_total_size_of_aggregate_states, 16),
+ _offsets_of_aggregate_states.data());
+ for (auto cid = _tablet_schema->num_key_columns(); cid < _num_columns; cid++) {
+ auto* col_ptr = mutable_block.mutable_columns()[cid].get();
+ auto* data = row->agg_places(cid);
+ _agg_functions[cid]->create(data);
+ _agg_functions[cid]->add(data, const_cast<const doris::vectorized::IColumn**>(&col_ptr),
+ row->_row_pos, _arena.get());
+ }
+}
+void MemTable::_clear_row_agg(RowInBlock* row) {
Review Comment:
warning: method '_clear_row_agg' can be made const [readability-make-member-function-const]
be/src/olap/memtable.h:259:
```diff
- void _clear_row_agg(RowInBlock* row);
+ void _clear_row_agg(RowInBlock* row) const;
```
```suggestion
void MemTable::_clear_row_agg(RowInBlock* row) const {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix incorrect result when there are insert after delete for same key in one load [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring commented on code in PR #41701:
URL: https://github.com/apache/doris/pull/41701#discussion_r1798677787
##########
be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:
##########
@@ -849,32 +906,44 @@ Status VerticalSegmentWriter::_merge_rows_for_sequence_column(
RETURN_IF_ERROR(_generate_encoded_default_seq_value(
*_tablet_schema, *_opts.rowset_ctx->partial_update_info, &encoded_default_seq_value));
- for (size_t block_pos = data.row_pos; block_pos < data.row_pos + data.num_rows; block_pos++) {
- size_t delta_pos = block_pos - data.row_pos;
- auto& skip_bitmap = skip_bitmaps->at(block_pos);
- std::string key = _full_encode_keys(key_columns, delta_pos);
- bool row_has_sequence_col = (!skip_bitmap.contains(seq_col_unique_id));
- Status st;
- if (delta_pos > 0 && previous_key == key) {
- DCHECK(previous_has_seq_col == !row_has_sequence_col);
- ++duplicate_keys;
- RowLocation loc;
- RowsetSharedPtr rowset;
- size_t rid_missing_seq {};
- size_t rid_with_seq {};
- if (row_has_sequence_col) {
- rid_missing_seq = block_pos - 1;
- rid_with_seq = block_pos;
+ // batched_rows store rows with the same key that have different pattern:
+ // 0 -> row without sequence col and with delete sign
+ // 1 -> row without sequence col and without delete sign
+ // 2 -> row has sequence col with delete sign
+ // 3 -> row has sequence col and without delete sign
+ std::array<int64_t, 4> batched_rows {};
+ batched_rows.fill(-1);
+ bool has_same_rows {false};
+ auto get_idx = [](bool with_seq_col, bool has_delete_sign) {
+ if (!with_seq_col) {
+ if (has_delete_sign) {
+ return 0;
+ } else {
+ return 1;
+ }
Review Comment:
return has_delete_sign ? 0 : 1.
##########
be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:
##########
@@ -849,32 +906,44 @@ Status VerticalSegmentWriter::_merge_rows_for_sequence_column(
RETURN_IF_ERROR(_generate_encoded_default_seq_value(
*_tablet_schema, *_opts.rowset_ctx->partial_update_info, &encoded_default_seq_value));
- for (size_t block_pos = data.row_pos; block_pos < data.row_pos + data.num_rows; block_pos++) {
- size_t delta_pos = block_pos - data.row_pos;
- auto& skip_bitmap = skip_bitmaps->at(block_pos);
- std::string key = _full_encode_keys(key_columns, delta_pos);
- bool row_has_sequence_col = (!skip_bitmap.contains(seq_col_unique_id));
- Status st;
- if (delta_pos > 0 && previous_key == key) {
- DCHECK(previous_has_seq_col == !row_has_sequence_col);
- ++duplicate_keys;
- RowLocation loc;
- RowsetSharedPtr rowset;
- size_t rid_missing_seq {};
- size_t rid_with_seq {};
- if (row_has_sequence_col) {
- rid_missing_seq = block_pos - 1;
- rid_with_seq = block_pos;
+ // batched_rows store rows with the same key that have different pattern:
+ // 0 -> row without sequence col and with delete sign
+ // 1 -> row without sequence col and without delete sign
+ // 2 -> row has sequence col with delete sign
+ // 3 -> row has sequence col and without delete sign
+ std::array<int64_t, 4> batched_rows {};
+ batched_rows.fill(-1);
+ bool has_same_rows {false};
+ auto get_idx = [](bool with_seq_col, bool has_delete_sign) {
+ if (!with_seq_col) {
+ if (has_delete_sign) {
+ return 0;
+ } else {
+ return 1;
+ }
+ } else {
+ if (has_delete_sign) {
+ return 2;
} else {
- rid_missing_seq = block_pos;
- rid_with_seq = block_pos - 1;
+ return 3;
}
+ }
+ };
+
+ auto find_rows_to_filter = [&](const std::string& key) {
+ bool has_row_with_seq_col = (batched_rows[0] != -1 || batched_rows[1] != -1);
+ bool has_row_without_seq_col = (batched_rows[2] != -1 || batched_rows[3] != -1);
+ if (has_row_with_seq_col && has_row_without_seq_col) {
Review Comment:
add comment to explain when the if is false.
##########
be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:
##########
@@ -849,32 +906,44 @@ Status VerticalSegmentWriter::_merge_rows_for_sequence_column(
RETURN_IF_ERROR(_generate_encoded_default_seq_value(
*_tablet_schema, *_opts.rowset_ctx->partial_update_info, &encoded_default_seq_value));
- for (size_t block_pos = data.row_pos; block_pos < data.row_pos + data.num_rows; block_pos++) {
- size_t delta_pos = block_pos - data.row_pos;
- auto& skip_bitmap = skip_bitmaps->at(block_pos);
- std::string key = _full_encode_keys(key_columns, delta_pos);
- bool row_has_sequence_col = (!skip_bitmap.contains(seq_col_unique_id));
- Status st;
- if (delta_pos > 0 && previous_key == key) {
- DCHECK(previous_has_seq_col == !row_has_sequence_col);
- ++duplicate_keys;
- RowLocation loc;
- RowsetSharedPtr rowset;
- size_t rid_missing_seq {};
- size_t rid_with_seq {};
- if (row_has_sequence_col) {
- rid_missing_seq = block_pos - 1;
- rid_with_seq = block_pos;
+ // batched_rows store rows with the same key that have different pattern:
+ // 0 -> row without sequence col and with delete sign
+ // 1 -> row without sequence col and without delete sign
+ // 2 -> row has sequence col with delete sign
+ // 3 -> row has sequence col and without delete sign
+ std::array<int64_t, 4> batched_rows {};
+ batched_rows.fill(-1);
+ bool has_same_rows {false};
+ auto get_idx = [](bool with_seq_col, bool has_delete_sign) {
+ if (!with_seq_col) {
+ if (has_delete_sign) {
+ return 0;
+ } else {
+ return 1;
+ }
+ } else {
+ if (has_delete_sign) {
+ return 2;
} else {
- rid_missing_seq = block_pos;
- rid_with_seq = block_pos - 1;
+ return 3;
}
Review Comment:
return has_delete_sign ? 2 : 3;
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2496987739
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2497643284
run p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2556318340
<details>
<summary>TPC-H: <b>Total hot run time: 40450 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c9f2159f3b4ce84a1b67858cefd8b3ceced09e37, data reload: false
------ Round 1 ----------------------------------
q1 17620 7670 7350 7350
q2 2059 188 189 188
q3 10620 1078 1258 1078
q4 10568 735 712 712
q5 7627 2845 2771 2771
q6 238 152 149 149
q7 1007 659 619 619
q8 9256 1962 1919 1919
q9 6624 6525 6463 6463
q10 7047 2351 2388 2351
q11 474 273 259 259
q12 431 233 227 227
q13 17779 3004 2965 2965
q14 243 218 228 218
q15 582 514 507 507
q16 660 587 583 583
q17 1020 519 542 519
q18 7406 6824 6872 6824
q19 1353 1089 1029 1029
q20 464 184 187 184
q21 4062 3372 3227 3227
q22 395 334 308 308
Total cold run time: 107535 ms
Total hot run time: 40450 ms
----- Round 2, with runtime_filter_mode=off -----
q1 7251 7271 7327 7271
q2 336 239 236 236
q3 2953 3031 2978 2978
q4 2085 1864 1834 1834
q5 5684 5655 5667 5655
q6 229 147 151 147
q7 2305 1790 1852 1790
q8 3439 3579 3547 3547
q9 9002 9033 8993 8993
q10 3576 3569 3524 3524
q11 604 519 500 500
q12 805 647 584 584
q13 12560 3196 3190 3190
q14 316 289 274 274
q15 563 518 500 500
q16 685 665 650 650
q17 1864 1635 1634 1634
q18 8387 7668 7494 7494
q19 1732 1567 1591 1567
q20 2116 1903 1875 1875
q21 5772 5653 5433 5433
q22 671 582 594 582
Total cold run time: 72935 ms
Total hot run time: 60258 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2642216820
<details>
<summary>TPC-H: <b>Total hot run time: 31613 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ebcaf82a4f4c0679d1c2fcdb77d2a5355d5a12b6, data reload: false
------ Round 1 ----------------------------------
q1 17643 5225 5075 5075
q2 2071 311 168 168
q3 10397 1306 752 752
q4 10205 1046 538 538
q5 7535 2452 2365 2365
q6 184 164 135 135
q7 909 735 593 593
q8 9314 1308 1086 1086
q9 4942 4846 4643 4643
q10 6840 2310 1909 1909
q11 479 280 249 249
q12 340 360 225 225
q13 17768 3688 3077 3077
q14 248 226 225 225
q15 520 483 467 467
q16 627 617 590 590
q17 574 880 336 336
q18 6546 6384 6192 6192
q19 1754 1104 541 541
q20 320 321 184 184
q21 2660 2168 1960 1960
q22 375 329 303 303
Total cold run time: 102251 ms
Total hot run time: 31613 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5200 5129 5149 5129
q2 236 327 231 231
q3 2159 2699 2318 2318
q4 1492 1891 1397 1397
q5 4252 4152 4198 4152
q6 215 167 124 124
q7 1861 1807 1745 1745
q8 2622 2651 2571 2571
q9 7240 7234 7219 7219
q10 3052 3200 2810 2810
q11 572 516 526 516
q12 702 738 614 614
q13 3473 3937 3298 3298
q14 298 304 278 278
q15 507 467 451 451
q16 657 701 662 662
q17 1155 1609 1384 1384
q18 7661 7412 7363 7363
q19 799 789 819 789
q20 1975 2023 1883 1883
q21 5437 5107 4787 4787
q22 621 604 534 534
Total cold run time: 52186 ms
Total hot run time: 50255 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2644553350
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-3059920519
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-3060891480
run p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-3061475448
# Cloud UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/a5f0e6dc17faa644a55c5770a94ae939582a5fb0_a5f0e6dc17faa644a55c5770a94ae939582a5fb0_cloud/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/a5f0e6dc17faa644a55c5770a94ae939582a5fb0_a5f0e6dc17faa644a55c5770a94ae939582a5fb0_cloud/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 79.77% (1270/1592) |
| Line Coverage | 65.41% (21296/32559) |
| Region Coverage | 66.89% (10779/16114) |
| Branch Coverage | 56.55% (5679/10042) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-3056430554
<details>
<summary>TPC-DS: <b>Total hot run time: 185329 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e9db8846be973f1e4fe4bbff061ba3fe85bcf0e9, data reload: false
query1 1003 399 398 398
query2 6515 1743 1740 1740
query3 6742 211 213 211
query4 26663 23876 23244 23244
query5 4362 571 423 423
query6 307 223 195 195
query7 4644 497 295 295
query8 275 222 210 210
query9 8637 2675 2686 2675
query10 464 328 280 280
query11 15267 15017 14792 14792
query12 162 108 103 103
query13 1649 531 420 420
query14 8597 5662 5675 5662
query15 208 193 171 171
query16 7375 653 513 513
query17 1218 747 607 607
query18 2074 407 310 310
query19 194 180 160 160
query20 119 116 112 112
query21 215 129 113 113
query22 4213 4224 4198 4198
query23 34358 32992 33238 32992
query24 8435 2352 2387 2352
query25 530 492 395 395
query26 992 268 166 166
query27 2734 510 343 343
query28 4258 2157 2130 2130
query29 665 548 422 422
query30 288 215 186 186
query31 913 860 783 783
query32 70 63 60 60
query33 531 359 317 317
query34 783 841 541 541
query35 767 818 728 728
query36 949 964 872 872
query37 121 99 77 77
query38 4115 4043 4046 4043
query39 1492 1411 1376 1376
query40 200 119 106 106
query41 56 57 52 52
query42 126 111 102 102
query43 499 516 471 471
query44 1336 836 814 814
query45 173 165 160 160
query46 830 1028 615 615
query47 1759 1841 1766 1766
query48 380 414 323 323
query49 696 483 380 380
query50 619 690 417 417
query51 4144 4329 4120 4120
query52 109 104 95 95
query53 217 262 181 181
query54 571 575 492 492
query55 85 78 81 78
query56 319 326 285 285
query57 1196 1192 1132 1132
query58 265 249 255 249
query59 2626 2678 2604 2604
query60 317 317 316 316
query61 124 116 121 116
query62 801 723 662 662
query63 220 181 199 181
query64 3503 1003 701 701
query65 4269 4185 4165 4165
query66 1078 411 313 313
query67 16026 15754 15513 15513
query68 7917 882 529 529
query69 479 305 264 264
query70 1153 1155 1129 1129
query71 477 335 313 313
query72 5518 4710 4784 4710
query73 713 592 351 351
query74 9361 8814 9077 8814
query75 3837 3167 2714 2714
query76 3639 1221 716 716
query77 797 411 288 288
query78 10150 10413 9317 9317
query79 1986 827 587 587
query80 597 496 438 438
query81 478 256 224 224
query82 446 123 99 99
query83 248 255 244 244
query84 250 116 94 94
query85 773 357 341 341
query86 351 306 258 258
query87 4381 4384 4298 4298
query88 3487 2333 2302 2302
query89 382 328 288 288
query90 1923 209 200 200
query91 140 136 108 108
query92 76 57 62 57
query93 1486 958 595 595
query94 664 412 314 314
query95 373 296 287 287
query96 487 572 281 281
query97 2734 2755 2658 2658
query98 237 205 203 203
query99 1438 1422 1284 1284
Total cold run time: 272426 ms
Total hot run time: 185329 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-3060188206
# Cloud UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/17a611f76c1d78d8849a8607f4f0381afb3efce7_17a611f76c1d78d8849a8607f4f0381afb3efce7_cloud/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/17a611f76c1d78d8849a8607f4f0381afb3efce7_17a611f76c1d78d8849a8607f4f0381afb3efce7_cloud/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 81.91% (1250/1526) |
| Line Coverage | 67.64% (21341/31550) |
| Region Coverage | 67.22% (10585/15746) |
| Branch Coverage | 56.75% (5573/9820) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-3051746562
<details>
<summary>TPC-DS: <b>Total hot run time: 185825 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit da950276a9235965a5acc2d5ba77e2579c87ae70, data reload: false
query1 995 401 399 399
query2 6521 1700 1703 1700
query3 6740 207 207 207
query4 26399 23546 22965 22965
query5 4348 588 419 419
query6 303 210 197 197
query7 4621 483 292 292
query8 281 210 199 199
query9 8619 2658 2656 2656
query10 477 337 272 272
query11 15759 15134 14817 14817
query12 151 106 107 106
query13 1655 534 413 413
query14 8629 5920 5937 5920
query15 200 200 183 183
query16 7519 435 271 271
query17 1316 729 594 594
query18 2020 415 304 304
query19 195 187 208 187
query20 119 117 115 115
query21 209 122 103 103
query22 3999 3991 3947 3947
query23 34039 33074 32983 32983
query24 8443 2382 2349 2349
query25 515 493 395 395
query26 1242 273 145 145
query27 2735 505 342 342
query28 4286 2171 2138 2138
query29 750 561 444 444
query30 297 211 195 195
query31 932 841 764 764
query32 71 68 89 68
query33 555 324 293 293
query34 778 841 518 518
query35 625 681 564 564
query36 922 987 885 885
query37 111 104 77 77
query38 4150 4226 4108 4108
query39 1506 1412 1399 1399
query40 212 116 102 102
query41 55 60 51 51
query42 122 108 113 108
query43 513 500 484 484
query44 1298 819 807 807
query45 174 168 169 168
query46 827 1016 628 628
query47 1767 1766 1689 1689
query48 405 419 300 300
query49 732 466 397 397
query50 626 694 410 410
query51 4157 4148 4178 4148
query52 102 105 93 93
query53 226 250 186 186
query54 573 562 506 506
query55 84 84 86 84
query56 309 284 295 284
query57 1180 1168 1117 1117
query58 273 249 248 248
query59 2693 2732 2664 2664
query60 316 311 304 304
query61 122 124 124 124
query62 819 731 669 669
query63 224 192 184 184
query64 4335 1219 878 878
query65 4266 4203 4141 4141
query66 1115 422 313 313
query67 15666 15350 15241 15241
query68 7949 870 531 531
query69 509 304 267 267
query70 1206 1166 1116 1116
query71 450 326 319 319
query72 5788 4775 4997 4775
query73 718 665 353 353
query74 8898 9004 8960 8960
query75 3757 3181 2704 2704
query76 3619 1139 707 707
query77 783 376 284 284
query78 11095 11174 10377 10377
query79 1727 877 609 609
query80 650 514 434 434
query81 474 255 217 217
query82 183 132 94 94
query83 258 261 242 242
query84 247 110 91 91
query85 775 363 321 321
query86 384 300 294 294
query87 4457 4474 4355 4355
query88 3040 2311 2310 2310
query89 382 318 289 289
query90 1969 208 200 200
query91 156 141 114 114
query92 77 60 54 54
query93 1685 953 587 587
query94 632 306 193 193
query95 376 297 287 287
query96 487 548 281 281
query97 2676 2772 2714 2714
query98 233 212 205 205
query99 1334 1422 1280 1280
Total cold run time: 273197 ms
Total hot run time: 185825 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-3051321200
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix incorrect result when there are insert after delete for same key in one load [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #41701:
URL: https://github.com/apache/doris/pull/41701#discussion_r1796863285
##########
be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:
##########
@@ -831,15 +885,18 @@ Status VerticalSegmentWriter::_generate_flexible_read_plan(
Status VerticalSegmentWriter::_merge_rows_for_sequence_column(
Review Comment:
warning: function '_merge_rows_for_sequence_column' has cognitive complexity of 95 (threshold 50) [readability-function-cognitive-complexity]
```cpp
Status VerticalSegmentWriter::_merge_rows_for_sequence_column(
^
```
<details>
<summary>Additional context</summary>
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:905:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
RETURN_IF_ERROR(_generate_encoded_default_seq_value(
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:905:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(_generate_encoded_default_seq_value(
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:916:** nesting level increased to 1
```cpp
auto get_idx = [](bool with_seq_col, bool has_delete_sign) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:917:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (!with_seq_col) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:918:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (has_delete_sign) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:920:** +1, nesting level increased to 3
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:923:** +1, nesting level increased to 2
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:924:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (has_delete_sign) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:926:** +1, nesting level increased to 3
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:931:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
for (size_t block_pos = data.row_pos; block_pos < data.row_pos + data.num_rows; block_pos++) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:937:** +1
```cpp
(!skip_bitmap.contains(delete_sign_col_unique_id) && delete_signs[block_pos] != 0);
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:938:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (delta_pos > 0 && previous_key == key) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:938:** +1
```cpp
if (delta_pos > 0 && previous_key == key) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:939:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (!has_same_rows) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:945:** +1, nesting level increased to 2
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:946:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (has_same_rows) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:949:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
if (has_row_with_seq_col && has_row_without_seq_col) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:949:** +1
```cpp
if (has_row_with_seq_col && has_row_without_seq_col) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:960:** +1
```cpp
DCHECK(st.is<KEY_NOT_FOUND>() || st.ok());
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:962:** +5, including nesting penalty of 4, nesting level increased to 5
```cpp
if (st.is<KEY_NOT_FOUND>()) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:964:** +1, nesting level increased to 5
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:969:** +5, including nesting penalty of 4, nesting level increased to 5
```cpp
if (batched_rows[2] != -1) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:973:** +5, including nesting penalty of 4, nesting level increased to 5
```cpp
if (batched_rows[3] != -1) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:978:** nesting level increased to 5
```cpp
auto remove_rows_without_seq = [&]() {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:979:** +6, including nesting penalty of 5, nesting level increased to 6
```cpp
if (batched_rows[0] != -1) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:983:** +6, including nesting penalty of 5, nesting level increased to 6
```cpp
if (batched_rows[1] != -1) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:990:** +5, including nesting penalty of 4, nesting level increased to 5
```cpp
if (batched_rows[2] != -1 && batched_rows[3] != -1) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:991:** +6, including nesting penalty of 5, nesting level increased to 6
```cpp
if (previous_seq_slice.compare(
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:994:** +1, nesting level increased to 6
```cpp
} else if (previous_seq_slice.compare(Slice {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:999:** +1, nesting level increased to 6
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1005:** +1, nesting level increased to 5
```cpp
} else if (batched_rows[2] != -1) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1006:** +6, including nesting penalty of 5, nesting level increased to 6
```cpp
if (previous_seq_slice.compare(
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1009:** +1, nesting level increased to 6
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1013:** +1, nesting level increased to 5
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1014:** +6, including nesting penalty of 5, nesting level increased to 6
```cpp
if (previous_seq_slice.compare(
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1017:** +1, nesting level increased to 6
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1031:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
if (duplicate_keys > 0) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1032:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(_filter_block_for_flexible_partial_update(
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1032:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(_filter_block_for_flexible_partial_update(
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix incorrect result when there are insert after delete for same key in one load [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2408485957
<details>
<summary>ClickBench: <b>Total hot run time: 32.6 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 79fe26c74dbc3442b2b7b76045ec77054fcb9a10, data reload: false
query1 0.04 0.03 0.03
query2 0.07 0.03 0.04
query3 0.23 0.07 0.07
query4 1.65 0.10 0.10
query5 0.52 0.50 0.50
query6 1.14 0.73 0.72
query7 0.02 0.02 0.02
query8 0.04 0.05 0.03
query9 0.56 0.52 0.51
query10 0.56 0.54 0.56
query11 0.14 0.11 0.11
query12 0.14 0.10 0.10
query13 0.62 0.60 0.59
query14 2.71 2.72 2.85
query15 0.92 0.82 0.85
query16 0.38 0.38 0.38
query17 0.96 1.04 1.00
query18 0.19 0.20 0.19
query19 1.92 1.75 1.89
query20 0.01 0.02 0.01
query21 15.35 0.57 0.55
query22 2.99 2.21 1.95
query23 17.01 0.92 0.79
query24 2.77 2.07 1.00
query25 0.28 0.08 0.12
query26 0.50 0.15 0.13
query27 0.05 0.04 0.04
query28 10.10 1.09 1.08
query29 12.59 3.24 3.19
query30 0.24 0.06 0.05
query31 2.89 0.37 0.38
query32 3.29 0.46 0.47
query33 3.01 3.00 3.02
query34 16.68 4.51 4.45
query35 4.51 4.46 4.50
query36 0.65 0.48 0.48
query37 0.08 0.06 0.06
query38 0.05 0.03 0.04
query39 0.03 0.02 0.03
query40 0.16 0.13 0.12
query41 0.08 0.02 0.02
query42 0.03 0.02 0.02
query43 0.04 0.03 0.03
Total cold run time: 106.2 s
Total hot run time: 32.6 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix incorrect result when there are insert after delete for same key in one load [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring commented on code in PR #41701:
URL: https://github.com/apache/doris/pull/41701#discussion_r1798676405
##########
be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:
##########
@@ -883,45 +952,158 @@ Status VerticalSegmentWriter::_merge_rows_for_sequence_column(
_rsid_to_rowset.emplace(rowset->rowset_id(), rowset);
previous_seq_slice = Slice {previous_encoded_seq_value};
}
- std::string cur_encoded_seq_value {};
- _encode_seq_column(seq_column, rid_with_seq, &cur_encoded_seq_value);
+ if (batched_rows[2] != -1) {
+ _encode_seq_column(seq_column, batched_rows[2],
+ &row_with_delete_sign_encoded_seq_value);
+ }
+ if (batched_rows[3] != -1) {
+ _encode_seq_column(seq_column, batched_rows[3],
+ &row_without_delete_sign_encoded_seq_value);
+ }
+
+ auto remove_rows_without_seq = [&]() {
+ if (batched_rows[0] != -1) {
+ filter_map[batched_rows[0]] = 0;
+ ++duplicate_keys;
+ }
+ if (batched_rows[1] != -1) {
+ filter_map[batched_rows[1]] = 0;
+ ++duplicate_keys;
+ }
+ };
+
// the encoded value is order-preserving, so we can use Slice::compare() to compare them
- int res = previous_seq_slice.compare(Slice {cur_encoded_seq_value});
- VLOG_DEBUG << fmt::format(
- "VerticalSegmentWriter::_merge_rows_for_sequence_column: rid_with_seq={}, "
- "rid_missing_seq={}, res={}",
- rid_with_seq, rid_missing_seq, res);
- if (res > 0) {
- filter_map[rid_with_seq] = 0;
- } else if (res < 0) {
- filter_map[rid_missing_seq] = 0;
+ if (batched_rows[2] != -1 && batched_rows[3] != -1) {
+ if (previous_seq_slice.compare(Slice {row_with_delete_sign_encoded_seq_value}) <=
+ 0) {
+ remove_rows_without_seq();
+ } else if (previous_seq_slice.compare(
+ Slice {row_without_delete_sign_encoded_seq_value}) <= 0) {
+ remove_rows_without_seq();
+ filter_map[batched_rows[2]] = 0;
+ ++duplicate_keys;
+ } else {
+ filter_map[batched_rows[2]] = 0;
+ ++duplicate_keys;
+ filter_map[batched_rows[3]] = 0;
+ ++duplicate_keys;
+ }
+ } else if (batched_rows[2] != -1) {
+ if (previous_seq_slice.compare(Slice {row_with_delete_sign_encoded_seq_value}) <=
+ 0) {
+ remove_rows_without_seq();
+ } else {
+ filter_map[batched_rows[2]] = 0;
+ ++duplicate_keys;
+ }
} else {
- filter_map[std::min(rid_with_seq, rid_missing_seq)] = 0;
+ if (previous_seq_slice.compare(Slice {row_without_delete_sign_encoded_seq_value}) <=
+ 0) {
+ remove_rows_without_seq();
+ } else {
+ filter_map[batched_rows[3]] = 0;
+ ++duplicate_keys;
+ }
+ }
+ }
+ has_same_rows = false;
+ batched_rows.fill(-1);
+ };
+
+ for (size_t block_pos = data.row_pos; block_pos < data.row_pos + data.num_rows; block_pos++) {
+ size_t delta_pos = block_pos - data.row_pos;
+ auto& skip_bitmap = skip_bitmaps->at(block_pos);
+ std::string key = _full_encode_keys(key_columns, delta_pos);
+ bool row_has_sequence_col = (!skip_bitmap.contains(seq_col_unique_id));
+ bool row_has_delete_sign =
+ (!skip_bitmap.contains(delete_sign_col_unique_id) && delete_signs[block_pos] != 0);
+ VLOG_DEBUG << fmt::format("block_pos={}, row_has_sequence_col={}, row_has_delete_sign={}",
+ block_pos, row_has_sequence_col, row_has_delete_sign);
+ if (delta_pos > 0 && previous_key == key) {
+ if (!has_same_rows) {
+ batched_rows[get_idx(previous_has_seq_col, previous_has_delete_sign)] =
+ block_pos - 1;
+ has_same_rows = true;
+ }
+ batched_rows[get_idx(row_has_sequence_col, row_has_delete_sign)] = block_pos;
+ } else {
+ if (has_same_rows) {
+ find_rows_to_filter(previous_key);
}
}
previous_key = std::move(key);
previous_has_seq_col = row_has_sequence_col;
+ previous_has_delete_sign = row_has_delete_sign;
+ }
+ if (has_same_rows) {
+ find_rows_to_filter(previous_key);
}
if (duplicate_keys > 0) {
- auto num_cols = data.block->columns();
- auto* block = const_cast<vectorized::Block*>(data.block);
- block->insert({std::move(filter_column), std::make_shared<vectorized::DataTypeUInt8>(),
- "__dup_key_filter_col__"});
- RETURN_IF_ERROR(vectorized::Block::filter_block(block, num_cols, num_cols));
- int merged_rows = data.num_rows - block->rows();
- VLOG_DEBUG << fmt::format(
- "VerticalSegmentWriter::_merge_rows_for_sequence_column after filter: "
- "data.block:{}\n",
- data.block->dump_data());
- if (duplicate_keys != merged_rows) {
- auto msg = fmt::format(
- "duplicate_keys != merged_rows, duplicate_keys={}, merged_rows={}, "
- "num_rows={}, mutable_block->rows()={}",
- duplicate_keys, merged_rows, data.num_rows, block->rows());
- DCHECK(false) << msg;
- return Status::InternalError<false>(msg);
+ RETURN_IF_ERROR(_filter_block_for_flexible_partial_update(
+ data, std::move(filter_column), duplicate_keys, "__dup_key_filter_col__"));
+ }
+ return Status::OK();
+}
+
+Status VerticalSegmentWriter::_merge_rows_for_insert_after_delete(
+ RowsInBlock& data, std::vector<BitmapValue>* skip_bitmaps,
+ const std::vector<vectorized::IOlapColumnDataAccessor*>& key_columns,
+ vectorized::IOlapColumnDataAccessor* seq_column, const signed char* delete_signs,
+ const std::vector<RowsetSharedPtr>& specified_rowsets,
+ std::vector<std::unique_ptr<SegmentCacheHandle>>& segment_caches) {
+ // TODO(bobhan1): handle sequence column later
+ VLOG_DEBUG << fmt::format(
+ "VerticalSegmentWriter::_merge_rows_for_insert_after_delete enter: data.block:{}\n",
+ data.block->dump_data());
+ auto filter_column = vectorized::ColumnUInt8::create(data.num_rows, 1);
+ auto* __restrict filter_map = filter_column->get_data().data();
+ std::string previous_key {};
+ bool previous_has_delete_sign {false};
+ int duplicate_rows {0};
+ int32_t delete_sign_col_unique_id =
+ _tablet_schema->column(_tablet_schema->delete_sign_idx()).unique_id();
+ for (size_t block_pos = data.row_pos; block_pos < data.row_pos + data.num_rows; block_pos++) {
+ size_t delta_pos = block_pos - data.row_pos;
+ auto& skip_bitmap = skip_bitmaps->at(block_pos);
+ std::string key = _full_encode_keys(key_columns, delta_pos);
+ bool have_delete_sign =
+ (!skip_bitmap.contains(delete_sign_col_unique_id) && delete_signs[block_pos] != 0);
+ if (delta_pos > 0 && previous_key == key) {
+ // !!ATTENTION!!: We can only remove the row with delete sign if there is a insert with the same key after this row.
+ // If there is only a row with delete sign, we should keep it and can't remove it from block, because
+ // compaction will not use the delete bitmap when reading data. So there may still be rows with delete sign
+ // in later process
+ DCHECK(previous_has_delete_sign);
+ DCHECK(!have_delete_sign);
+ ++duplicate_rows;
+ RowLocation loc;
+ RowsetSharedPtr rowset;
+ Status st = _tablet->lookup_row_key(key, _tablet_schema.get(), false, specified_rowsets,
+ &loc, _mow_context->max_version, segment_caches,
+ &rowset, true);
+ DCHECK(st.is<KEY_NOT_FOUND>() || st.ok());
+
+ Slice previous_seq_slice {};
+ if (st.ok()) {
+ // delete the existing row
+ _mow_context->delete_bitmap->add(
+ {loc.rowset_id, loc.segment_id, DeleteBitmap::TEMP_VERSION_COMMON},
+ loc.row_id);
+ VLOG_DEBUG << fmt::format(
+ "_merge_rows_for_insert_after_delete: delete existing row, rowset_id={}, "
+ "segment_id={}, row_id={}",
+ loc.rowset_id.to_string(), loc.segment_id, loc.row_id);
+ }
+ // and remove the row with delete sign from the current block
+ filter_map[block_pos - 1] = 0;
}
- data.num_rows = block->rows();
+ previous_has_delete_sign = have_delete_sign;
+ previous_key = std::move(key);
+ }
+ if (duplicate_rows > 0) {
+ RETURN_IF_ERROR(_filter_block_for_flexible_partial_update(
Review Comment:
++duplicate_rows above is just for checking here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix incorrect result when there are insert after delete for same key in one load [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2413878240
<details>
<summary>ClickBench: <b>Total hot run time: 32.99 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 99e53bd3da17aa9c7e310bf4b8d91beb7a81b396, data reload: false
query1 0.04 0.04 0.03
query2 0.06 0.03 0.03
query3 0.24 0.06 0.07
query4 1.64 0.11 0.10
query5 0.52 0.50 0.51
query6 1.13 0.74 0.74
query7 0.02 0.02 0.01
query8 0.04 0.03 0.04
query9 0.57 0.50 0.50
query10 0.55 0.54 0.54
query11 0.14 0.11 0.11
query12 0.13 0.11 0.11
query13 0.61 0.59 0.59
query14 2.85 2.70 2.75
query15 0.90 0.83 0.83
query16 0.39 0.38 0.39
query17 1.09 1.04 1.06
query18 0.20 0.20 0.19
query19 1.87 1.90 1.99
query20 0.01 0.01 0.01
query21 15.38 0.60 0.60
query22 2.68 2.10 2.18
query23 17.11 1.10 0.77
query24 3.08 0.97 0.98
query25 0.26 0.13 0.10
query26 0.38 0.13 0.14
query27 0.05 0.04 0.04
query28 10.95 1.09 1.06
query29 12.56 3.23 3.24
query30 0.24 0.06 0.06
query31 2.88 0.38 0.38
query32 3.27 0.47 0.46
query33 3.00 2.98 3.01
query34 17.24 4.46 4.52
query35 4.56 4.57 4.48
query36 0.63 0.48 0.49
query37 0.08 0.06 0.05
query38 0.05 0.04 0.03
query39 0.03 0.02 0.02
query40 0.16 0.12 0.12
query41 0.07 0.02 0.02
query42 0.03 0.02 0.02
query43 0.04 0.03 0.03
Total cold run time: 107.73 s
Total hot run time: 32.99 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix incorrect result when there are insert after delete for same key in one load [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #41701:
URL: https://github.com/apache/doris/pull/41701#discussion_r1806349791
##########
be/src/olap/partial_update_info.cpp:
##########
@@ -658,4 +664,422 @@ Status FlexibleReadPlan::fill_non_primary_key_columns_for_row_store(
return Status::OK();
}
+BlockAggregator::BlockAggregator(segment_v2::VerticalSegmentWriter& vertical_segment_writer)
+ : _writer(vertical_segment_writer), _tablet_schema(*_writer._tablet_schema) {}
+
+void BlockAggregator::merge_one_row(vectorized::MutableBlock& dst_block,
+ vectorized::Block* src_block, int64_t rid,
+ BitmapValue& skip_bitmap) {
+ for (size_t cid {_tablet_schema.num_key_columns()}; cid < _tablet_schema.num_columns(); cid++) {
+ if (cid == _tablet_schema.skip_bitmap_col_idx()) {
+ auto& cur_skip_bitmap =
+ assert_cast<vectorized::ColumnBitmap*>(dst_block.mutable_columns()[cid].get())
+ ->get_data()
+ .back();
+ const auto& new_row_skip_bitmap =
+ assert_cast<vectorized::ColumnBitmap*>(
+ src_block->get_by_position(cid).column->assume_mutable().get())
+ ->get_data()[rid];
+ cur_skip_bitmap &= new_row_skip_bitmap;
+ continue;
+ }
+ if (!skip_bitmap.contains(_tablet_schema.column(cid).unique_id())) {
+ dst_block.mutable_columns()[cid]->pop_back(1);
+ dst_block.mutable_columns()[cid]->insert_from(*src_block->get_by_position(cid).column,
+ rid);
+ }
+ }
+ VLOG_DEBUG << fmt::format("merge a row, after merge, output_block.rows()={}, state: {}",
+ dst_block.rows(), _state.to_string());
+}
+
+void BlockAggregator::append_one_row(vectorized::MutableBlock& dst_block,
+ vectorized::Block* src_block, int64_t rid) {
+ dst_block.add_row(src_block, rid);
+ _state.rows++;
+ VLOG_DEBUG << fmt::format("append a new row, after append, output_block.rows()={}, state: {}",
+ dst_block.rows(), _state.to_string());
+}
+
+void BlockAggregator::remove_last_n_rows(vectorized::MutableBlock& dst_block, int n) {
+ if (n > 0) {
+ for (size_t cid {0}; cid < _tablet_schema.num_columns(); cid++) {
+ DCHECK_GE(dst_block.mutable_columns()[cid]->size(), n);
+ dst_block.mutable_columns()[cid]->pop_back(n);
+ }
+ }
+}
+
+void BlockAggregator::append_or_merge_row(vectorized::MutableBlock& dst_block,
+ vectorized::Block* src_block, int64_t rid,
+ BitmapValue& skip_bitmap, bool have_delete_sign) {
+ if (have_delete_sign) {
+ // remove all the previous batched rows
+ remove_last_n_rows(dst_block, _state.rows);
+ _state.rows = 0;
+ _state.has_row_with_delete_sign = true;
+
+ append_one_row(dst_block, src_block, rid);
+ } else {
+ if (_state.should_merge()) {
+ merge_one_row(dst_block, src_block, rid, skip_bitmap);
+ } else {
+ append_one_row(dst_block, src_block, rid);
+ }
+ }
+};
+
+Status BlockAggregator::aggregate_rows(
Review Comment:
warning: function 'aggregate_rows' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
Status BlockAggregator::aggregate_rows(
^
```
<details>
<summary>Additional context</summary>
**be/src/olap/partial_update_info.cpp:731:** 82 lines including whitespace and comments (threshold 80)
```cpp
Status BlockAggregator::aggregate_rows(
^
```
</details>
##########
be/src/olap/partial_update_info.cpp:
##########
@@ -658,4 +664,422 @@ Status FlexibleReadPlan::fill_non_primary_key_columns_for_row_store(
return Status::OK();
}
+BlockAggregator::BlockAggregator(segment_v2::VerticalSegmentWriter& vertical_segment_writer)
+ : _writer(vertical_segment_writer), _tablet_schema(*_writer._tablet_schema) {}
+
+void BlockAggregator::merge_one_row(vectorized::MutableBlock& dst_block,
+ vectorized::Block* src_block, int64_t rid,
+ BitmapValue& skip_bitmap) {
+ for (size_t cid {_tablet_schema.num_key_columns()}; cid < _tablet_schema.num_columns(); cid++) {
+ if (cid == _tablet_schema.skip_bitmap_col_idx()) {
+ auto& cur_skip_bitmap =
+ assert_cast<vectorized::ColumnBitmap*>(dst_block.mutable_columns()[cid].get())
+ ->get_data()
+ .back();
+ const auto& new_row_skip_bitmap =
+ assert_cast<vectorized::ColumnBitmap*>(
+ src_block->get_by_position(cid).column->assume_mutable().get())
+ ->get_data()[rid];
+ cur_skip_bitmap &= new_row_skip_bitmap;
+ continue;
+ }
+ if (!skip_bitmap.contains(_tablet_schema.column(cid).unique_id())) {
+ dst_block.mutable_columns()[cid]->pop_back(1);
+ dst_block.mutable_columns()[cid]->insert_from(*src_block->get_by_position(cid).column,
+ rid);
+ }
+ }
+ VLOG_DEBUG << fmt::format("merge a row, after merge, output_block.rows()={}, state: {}",
+ dst_block.rows(), _state.to_string());
+}
+
+void BlockAggregator::append_one_row(vectorized::MutableBlock& dst_block,
+ vectorized::Block* src_block, int64_t rid) {
+ dst_block.add_row(src_block, rid);
+ _state.rows++;
+ VLOG_DEBUG << fmt::format("append a new row, after append, output_block.rows()={}, state: {}",
+ dst_block.rows(), _state.to_string());
+}
+
+void BlockAggregator::remove_last_n_rows(vectorized::MutableBlock& dst_block, int n) {
+ if (n > 0) {
+ for (size_t cid {0}; cid < _tablet_schema.num_columns(); cid++) {
+ DCHECK_GE(dst_block.mutable_columns()[cid]->size(), n);
+ dst_block.mutable_columns()[cid]->pop_back(n);
+ }
+ }
+}
+
+void BlockAggregator::append_or_merge_row(vectorized::MutableBlock& dst_block,
+ vectorized::Block* src_block, int64_t rid,
+ BitmapValue& skip_bitmap, bool have_delete_sign) {
+ if (have_delete_sign) {
+ // remove all the previous batched rows
+ remove_last_n_rows(dst_block, _state.rows);
+ _state.rows = 0;
+ _state.has_row_with_delete_sign = true;
+
+ append_one_row(dst_block, src_block, rid);
+ } else {
+ if (_state.should_merge()) {
+ merge_one_row(dst_block, src_block, rid, skip_bitmap);
+ } else {
+ append_one_row(dst_block, src_block, rid);
+ }
+ }
+};
+
+Status BlockAggregator::aggregate_rows(
+ vectorized::MutableBlock& output_block, vectorized::Block* block, int64_t start,
+ int64_t end, std::string key, std::vector<BitmapValue>* skip_bitmaps,
+ const signed char* delete_signs, vectorized::IOlapColumnDataAccessor* seq_column,
+ const std::vector<RowsetSharedPtr>& specified_rowsets,
+ std::vector<std::unique_ptr<SegmentCacheHandle>>& segment_caches) {
+ VLOG_DEBUG << fmt::format("merge rows in range=[{}-{})", start, end);
+ if (end - start == 1) {
+ output_block.add_row(block, start);
+ VLOG_DEBUG << fmt::format("append a row directly, rid={}", start);
+ return Status::OK();
+ }
+
+ auto seq_col_unique_id = _tablet_schema.column(_tablet_schema.sequence_col_idx()).unique_id();
+ auto delete_sign_col_unique_id =
+ _tablet_schema.column(_tablet_schema.delete_sign_idx()).unique_id();
+
+ _state.reset();
+
+ RowLocation loc;
+ RowsetSharedPtr rowset;
+ std::string previous_encoded_seq_value {};
+ Status st = _writer._tablet->lookup_row_key(
+ key, &_tablet_schema, false, specified_rowsets, &loc, _writer._mow_context->max_version,
+ segment_caches, &rowset, true, &previous_encoded_seq_value);
+ int64_t pos = start;
+ DCHECK(st.is<ErrorCode::KEY_NOT_FOUND>() || st.ok());
+
+ std::string cur_seq_val;
+ if (st.ok()) {
+ for (int64_t i {start}; i < end; i++) {
+ auto& skip_bitmap = skip_bitmaps->at(i);
+ bool row_has_sequence_col = (!skip_bitmap.contains(seq_col_unique_id));
+ // Find the first row that has larger sequence value than the existing row's
+ // or the first row that doesn't specify the sequence column.
+ // Discard all the rows before and begin to do aggregation from that row
+ if (!row_has_sequence_col) {
+ cur_seq_val = std::move(previous_encoded_seq_value);
+ pos = i;
+ break;
+ } else {
+ std::string seq_val {};
+ _writer._encode_seq_column(seq_column, i, &seq_val);
+ if (Slice {seq_val}.compare(Slice {previous_encoded_seq_value}) >= 0) {
+ cur_seq_val = std::move(seq_val);
+ pos = i;
+ break;
+ }
+ }
+ }
+ } else {
+ pos = start;
+ auto& skip_bitmap = skip_bitmaps->at(pos);
+ bool row_has_sequence_col = (!skip_bitmap.contains(seq_col_unique_id));
+ if (row_has_sequence_col) {
+ std::string seq_val {};
+ // for rows that don't specify seqeunce col, seq_val will be encoded to minial value
+ _writer._encode_seq_column(seq_column, pos, &seq_val);
+ cur_seq_val = std::move(seq_val);
+ } else {
+ cur_seq_val.clear();
+ RETURN_IF_ERROR(_writer._generate_encoded_default_seq_value(
+ _tablet_schema, *_writer._opts.rowset_ctx->partial_update_info, &cur_seq_val));
+ }
+ }
+ VLOG_DEBUG << fmt::format("start pos={}, output_block.rows()={}", pos, output_block.rows());
+
+ for (int64_t rid {pos}; rid < end; rid++) {
+ auto& skip_bitmap = skip_bitmaps->at(rid);
+ bool row_has_sequence_col = (!skip_bitmap.contains(seq_col_unique_id));
+ bool have_delete_sign =
+ (!skip_bitmap.contains(delete_sign_col_unique_id) && delete_signs[rid] != 0);
+ if (!row_has_sequence_col) {
+ append_or_merge_row(output_block, block, rid, skip_bitmap, have_delete_sign);
+ } else {
+ std::string seq_val {};
+ _writer._encode_seq_column(seq_column, rid, &seq_val);
+ if (Slice {seq_val}.compare(Slice {cur_seq_val}) >= 0) {
+ append_or_merge_row(output_block, block, rid, skip_bitmap, have_delete_sign);
+ cur_seq_val = std::move(seq_val);
+ } else {
+ VLOG_DEBUG << fmt::format(
+ "skip rid={} becasue its seq value is lower than the previous", rid);
+ }
+ }
+ }
+ return Status::OK();
+};
+
+Status BlockAggregator::aggregate_for_sequence_column(
+ vectorized::Block* block, size_t num_rows,
+ const std::vector<vectorized::IOlapColumnDataAccessor*>& key_columns,
+ vectorized::IOlapColumnDataAccessor* seq_column,
+ const std::vector<RowsetSharedPtr>& specified_rowsets,
+ std::vector<std::unique_ptr<SegmentCacheHandle>>& segment_caches) {
+ DCHECK_EQ(block->columns(), _tablet_schema.num_columns());
+ VLOG_DEBUG << fmt::format("BlockAggregator::aggregate_for_sequence_column enter: block:{}\n",
+ block->dump_data());
+ // the process logic here is the same as MemTable::_aggregate_for_flexible_partial_update_without_seq_col()
+ // after this function, there will be at most 2 rows for a specified key
+ std::vector<BitmapValue>* skip_bitmaps =
+ &(assert_cast<vectorized::ColumnBitmap*>(
+ block->get_by_position(_tablet_schema.skip_bitmap_col_idx())
+ .column->assume_mutable()
+ .get())
+ ->get_data());
+ const auto* delete_signs = BaseTablet::get_delete_sign_column_data(*block, num_rows);
+
+ auto filtered_block = _tablet_schema.create_block();
+ vectorized::MutableBlock output_block =
+ vectorized::MutableBlock::build_mutable_block(&filtered_block);
+
+ int same_key_rows {0};
+ std::string previous_key {};
+ for (size_t block_pos {0}; block_pos < num_rows; block_pos++) {
+ std::string key = _writer._full_encode_keys(key_columns, block_pos);
+ if (block_pos > 0 && previous_key == key) {
+ same_key_rows++;
+ } else {
+ if (same_key_rows > 0) {
+ RETURN_IF_ERROR(aggregate_rows(output_block, block, block_pos - same_key_rows,
+ block_pos, std::move(previous_key), skip_bitmaps,
+ delete_signs, seq_column, specified_rowsets,
+ segment_caches));
+ }
+ same_key_rows = 1;
+ }
+ previous_key = std::move(key);
+ }
+ if (same_key_rows > 0) {
+ RETURN_IF_ERROR(aggregate_rows(output_block, block, num_rows - same_key_rows, num_rows,
+ std::move(previous_key), skip_bitmaps, delete_signs,
+ seq_column, specified_rowsets, segment_caches));
+ }
+
+ block->swap(output_block.to_block());
+ VLOG_DEBUG << fmt::format(
+ "BlockAggregator::aggregate_for_sequence_column after: data.block:{}\n",
+ block->dump_data());
+ return Status::OK();
+}
+
+Status BlockAggregator::fill_sequence_column(vectorized::Block* block, size_t num_rows,
+ const FixedReadPlan& read_plan,
+ std::vector<BitmapValue>& skip_bitmaps) {
+ DCHECK(_tablet_schema.has_sequence_col());
+ std::vector<uint32_t> cids {static_cast<uint32_t>(_tablet_schema.sequence_col_idx())};
+ auto seq_col_unique_id = _tablet_schema.column(_tablet_schema.sequence_col_idx()).unique_id();
+
+ auto seq_col_block = _tablet_schema.create_block_by_cids(cids);
+ auto tmp_block = _tablet_schema.create_block_by_cids(cids);
+ std::map<uint32_t, uint32_t> read_index;
+ RETURN_IF_ERROR(read_plan.read_columns_by_plan(_tablet_schema, cids, _writer._rsid_to_rowset,
+ seq_col_block, &read_index));
+
+ auto new_seq_col_ptr = tmp_block.get_by_position(0).column->assume_mutable();
+ const auto& old_seq_col_ptr = *seq_col_block.get_by_position(0).column;
+ const auto& cur_seq_col_ptr = *block->get_by_position(_tablet_schema.sequence_col_idx()).column;
+ for (size_t block_pos {0}; block_pos < num_rows; block_pos++) {
+ if (read_index.contains(block_pos)) {
+ new_seq_col_ptr->insert_from(old_seq_col_ptr, read_index[block_pos]);
+ skip_bitmaps[block_pos].remove(seq_col_unique_id);
+ } else {
+ new_seq_col_ptr->insert_from(cur_seq_col_ptr, block_pos);
+ }
+ }
+ block->replace_by_position(_tablet_schema.sequence_col_idx(), std::move(new_seq_col_ptr));
+ return Status::OK();
+}
+
+Status BlockAggregator::aggregate_for_insert_after_delete(
Review Comment:
warning: function 'aggregate_for_insert_after_delete' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
Status BlockAggregator::aggregate_for_insert_after_delete(
^
```
<details>
<summary>Additional context</summary>
**be/src/olap/partial_update_info.cpp:901:** 89 lines including whitespace and comments (threshold 80)
```cpp
Status BlockAggregator::aggregate_for_insert_after_delete(
^
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix incorrect result when there are insert after delete for same key in one load [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2451474661
<details>
<summary>ClickBench: <b>Total hot run time: 33.22 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ffefc7377f05b2b19319afb73bc7375cd0370491, data reload: false
query1 0.03 0.03 0.03
query2 0.08 0.04 0.04
query3 0.22 0.06 0.05
query4 1.67 0.08 0.08
query5 0.41 0.38 0.41
query6 1.15 0.69 0.66
query7 0.02 0.02 0.02
query8 0.05 0.04 0.04
query9 0.56 0.49 0.50
query10 0.56 0.55 0.56
query11 0.17 0.12 0.12
query12 0.16 0.13 0.13
query13 0.61 0.61 0.60
query14 2.72 2.86 2.75
query15 0.93 0.83 0.85
query16 0.38 0.37 0.37
query17 1.06 1.04 1.06
query18 0.19 0.19 0.19
query19 1.99 1.88 1.97
query20 0.01 0.01 0.01
query21 15.37 0.66 0.65
query22 4.46 6.79 1.77
query23 18.24 1.40 1.30
query24 2.11 0.21 0.22
query25 0.14 0.08 0.08
query26 0.27 0.18 0.17
query27 0.09 0.08 0.08
query28 13.27 1.17 1.14
query29 12.63 3.47 3.42
query30 0.24 0.06 0.06
query31 2.86 0.42 0.40
query32 3.24 0.49 0.48
query33 2.98 3.06 3.04
query34 17.15 4.52 4.52
query35 4.66 4.64 4.57
query36 0.66 0.48 0.53
query37 0.20 0.16 0.15
query38 0.16 0.14 0.15
query39 0.06 0.04 0.04
query40 0.15 0.13 0.12
query41 0.10 0.05 0.05
query42 0.06 0.05 0.06
query43 0.04 0.04 0.05
Total cold run time: 112.11 s
Total hot run time: 33.22 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix incorrect result when there are insert after delete for same key in one load [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2443452031
<details>
<summary>TPC-DS: <b>Total hot run time: 191758 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2bde941d7f0c31261a07d9bdc7873b8f1521d792, data reload: false
query1 981 370 393 370
query2 6513 2114 2088 2088
query3 6787 212 224 212
query4 34379 23593 23691 23593
query5 4323 454 464 454
query6 260 167 163 163
query7 4604 296 287 287
query8 283 236 232 232
query9 9540 2641 2641 2641
query10 475 244 261 244
query11 18264 15453 15179 15179
query12 152 102 99 99
query13 1667 430 407 407
query14 9873 6897 7528 6897
query15 312 173 180 173
query16 8129 439 483 439
query17 1728 561 566 561
query18 2045 287 288 287
query19 369 159 162 159
query20 115 109 110 109
query21 205 104 101 101
query22 4590 4311 4239 4239
query23 34946 34134 34119 34119
query24 11084 2879 2857 2857
query25 663 407 419 407
query26 1387 162 165 162
query27 2783 282 286 282
query28 8110 2439 2401 2401
query29 916 422 420 420
query30 338 170 168 168
query31 1035 798 781 781
query32 91 55 59 55
query33 779 270 278 270
query34 981 519 510 510
query35 1029 884 893 884
query36 1092 951 963 951
query37 136 79 76 76
query38 4311 4319 4247 4247
query39 1491 1452 1389 1389
query40 274 103 101 101
query41 47 48 48 48
query42 114 104 99 99
query43 548 492 501 492
query44 1297 805 803 803
query45 181 170 166 166
query46 1152 707 691 691
query47 1929 1817 1850 1817
query48 419 326 316 316
query49 1172 411 402 402
query50 799 403 413 403
query51 7125 6958 7025 6958
query52 103 93 95 93
query53 261 186 185 185
query54 1247 425 418 418
query55 86 78 85 78
query56 277 251 241 241
query57 1318 1187 1205 1187
query58 243 209 238 209
query59 3385 3214 3210 3210
query60 277 247 239 239
query61 99 112 99 99
query62 863 666 696 666
query63 210 188 190 188
query64 5259 636 599 599
query65 3357 3195 3202 3195
query66 1436 315 309 309
query67 15971 15599 15629 15599
query68 4890 530 551 530
query69 447 291 258 258
query70 1162 1166 1161 1161
query71 334 254 263 254
query72 6386 4056 4023 4023
query73 765 366 363 363
query74 10312 8983 9109 8983
query75 3438 2670 2623 2623
query76 3073 1020 1007 1007
query77 395 273 267 267
query78 10663 9587 9524 9524
query79 1724 601 622 601
query80 1267 430 416 416
query81 586 251 245 245
query82 933 111 121 111
query83 210 139 140 139
query84 235 69 72 69
query85 1327 306 284 284
query86 439 288 288 288
query87 4803 4572 4551 4551
query88 3658 2191 2169 2169
query89 398 288 286 286
query90 2093 191 190 190
query91 136 97 102 97
query92 61 48 48 48
query93 1702 563 548 548
query94 973 292 306 292
query95 354 251 243 243
query96 630 292 295 292
query97 2870 2712 2686 2686
query98 214 191 193 191
query99 1557 1320 1295 1295
Total cold run time: 304250 ms
Total hot run time: 191758 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2475296444
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2490244575
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-3044961711
# BE UT Coverage Report
Increment line coverage `5.13% (35/682)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/269e246b2133708ac5567562989edaa162f27a63_269e246b2133708ac5567562989edaa162f27a63/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/269e246b2133708ac5567562989edaa162f27a63_269e246b2133708ac5567562989edaa162f27a63/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 57.28% (15499/27058) |
| Line Coverage | 46.24% (140851/304590) |
| Region Coverage | 45.49% (71208/156544) |
| Branch Coverage | 40.20% (37522/93346) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-3051398706
# Cloud UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/da950276a9235965a5acc2d5ba77e2579c87ae70_da950276a9235965a5acc2d5ba77e2579c87ae70_cloud/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/da950276a9235965a5acc2d5ba77e2579c87ae70_da950276a9235965a5acc2d5ba77e2579c87ae70_cloud/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 80.60% (1255/1557) |
| Line Coverage | 67.76% (21408/31596) |
| Region Coverage | 67.20% (10600/15774) |
| Branch Coverage | 56.74% (5566/9810) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-3060148493
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-3060831007
run cloud_p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #41701:
URL: https://github.com/apache/doris/pull/41701#discussion_r1835892343
##########
be/src/olap/partial_update_info.cpp:
##########
@@ -652,10 +739,440 @@
*old_value_block.get_by_position(i).column,
*block->get_by_position(cid).column, block_pos,
skip_bitmaps->at(block_pos).contains(col_uid),
+ tablet_schema.has_sequence_col()
+ ? !skip_bitmaps->at(block_pos).contains(seq_col_unique_id)
+ : false,
use_default_or_null_flag[idx], delete_sign_column_data, pos_in_old_block);
}
}
return Status::OK();
}
+BlockAggregator::BlockAggregator(segment_v2::VerticalSegmentWriter& vertical_segment_writer)
+ : _writer(vertical_segment_writer), _tablet_schema(*_writer._tablet_schema) {}
+
+void BlockAggregator::merge_one_row(vectorized::MutableBlock& dst_block,
+ vectorized::Block* src_block, int64_t rid,
+ BitmapValue& skip_bitmap) {
+ for (size_t cid {_tablet_schema.num_key_columns()}; cid < _tablet_schema.num_columns(); cid++) {
+ if (cid == _tablet_schema.skip_bitmap_col_idx()) {
+ auto& cur_skip_bitmap =
+ assert_cast<vectorized::ColumnBitmap*>(dst_block.mutable_columns()[cid].get())
+ ->get_data()
+ .back();
+ const auto& new_row_skip_bitmap =
+ assert_cast<vectorized::ColumnBitmap*>(
+ src_block->get_by_position(cid).column->assume_mutable().get())
+ ->get_data()[rid];
+ cur_skip_bitmap &= new_row_skip_bitmap;
+ continue;
+ }
+ if (!skip_bitmap.contains(_tablet_schema.column(cid).unique_id())) {
+ dst_block.mutable_columns()[cid]->pop_back(1);
+ dst_block.mutable_columns()[cid]->insert_from(*src_block->get_by_position(cid).column,
+ rid);
+ }
+ }
+ VLOG_DEBUG << fmt::format("merge a row, after merge, output_block.rows()={}, state: {}",
+ dst_block.rows(), _state.to_string());
+}
+
+void BlockAggregator::append_one_row(vectorized::MutableBlock& dst_block,
+ vectorized::Block* src_block, int64_t rid) {
+ dst_block.add_row(src_block, rid);
+ _state.rows++;
+ VLOG_DEBUG << fmt::format("append a new row, after append, output_block.rows()={}, state: {}",
+ dst_block.rows(), _state.to_string());
+}
+
+void BlockAggregator::remove_last_n_rows(vectorized::MutableBlock& dst_block, int n) {
+ if (n > 0) {
+ for (size_t cid {0}; cid < _tablet_schema.num_columns(); cid++) {
+ DCHECK_GE(dst_block.mutable_columns()[cid]->size(), n);
+ dst_block.mutable_columns()[cid]->pop_back(n);
+ }
+ }
+}
+
+void BlockAggregator::append_or_merge_row(vectorized::MutableBlock& dst_block,
+ vectorized::Block* src_block, int64_t rid,
+ BitmapValue& skip_bitmap, bool have_delete_sign) {
+ if (have_delete_sign) {
+ // remove all the previous batched rows
+ remove_last_n_rows(dst_block, _state.rows);
+ _state.rows = 0;
+ _state.has_row_with_delete_sign = true;
+
+ append_one_row(dst_block, src_block, rid);
+ } else {
+ if (_state.should_merge()) {
+ merge_one_row(dst_block, src_block, rid, skip_bitmap);
+ } else {
+ append_one_row(dst_block, src_block, rid);
+ }
+ }
+};
+
+Status BlockAggregator::aggregate_rows(
+ vectorized::MutableBlock& output_block, vectorized::Block* block, int64_t start,
+ int64_t end, std::string key, std::vector<BitmapValue>* skip_bitmaps,
+ const signed char* delete_signs, vectorized::IOlapColumnDataAccessor* seq_column,
+ const std::vector<RowsetSharedPtr>& specified_rowsets,
+ std::vector<std::unique_ptr<SegmentCacheHandle>>& segment_caches) {
+ VLOG_DEBUG << fmt::format("merge rows in range=[{}-{})", start, end);
+ if (end - start == 1) {
+ output_block.add_row(block, start);
+ VLOG_DEBUG << fmt::format("append a row directly, rid={}", start);
+ return Status::OK();
+ }
+
+ auto seq_col_unique_id = _tablet_schema.column(_tablet_schema.sequence_col_idx()).unique_id();
+ auto delete_sign_col_unique_id =
+ _tablet_schema.column(_tablet_schema.delete_sign_idx()).unique_id();
+
+ _state.reset();
+
+ RowLocation loc;
+ RowsetSharedPtr rowset;
+ std::string previous_encoded_seq_value {};
+ Status st = _writer._tablet->lookup_row_key(
+ key, &_tablet_schema, false, specified_rowsets, &loc, _writer._mow_context->max_version,
+ segment_caches, &rowset, true, &previous_encoded_seq_value);
+ int64_t pos = start;
+ bool is_expected_st = (st.is<ErrorCode::KEY_NOT_FOUND>() || st.ok());
+ DCHECK(is_expected_st || st.is<ErrorCode::MEM_LIMIT_EXCEEDED>())
+ << "[BlockAggregator::aggregate_rows] unexpected error status while lookup_row_key:"
+ << st;
+ if (!is_expected_st) {
+ return st;
+ }
+
+ std::string cur_seq_val;
+ if (st.ok()) {
+ for (pos = start; pos < end; pos++) {
+ auto& skip_bitmap = skip_bitmaps->at(pos);
+ bool row_has_sequence_col = (!skip_bitmap.contains(seq_col_unique_id));
+ // Discard all the rows whose seq value is smaller than previous_encoded_seq_value.
+ if (row_has_sequence_col) {
+ std::string seq_val {};
+ _writer._encode_seq_column(seq_column, pos, &seq_val);
+ if (Slice {seq_val}.compare(Slice {previous_encoded_seq_value}) < 0) {
+ continue;
+ }
+ cur_seq_val = std::move(seq_val);
+ break;
+ }
+ cur_seq_val = std::move(previous_encoded_seq_value);
+ break;
+ }
+ } else {
+ pos = start;
+ auto& skip_bitmap = skip_bitmaps->at(pos);
+ bool row_has_sequence_col = (!skip_bitmap.contains(seq_col_unique_id));
+ if (row_has_sequence_col) {
+ std::string seq_val {};
+ // for rows that don't specify seqeunce col, seq_val will be encoded to minial value
+ _writer._encode_seq_column(seq_column, pos, &seq_val);
+ cur_seq_val = std::move(seq_val);
+ } else {
+ cur_seq_val.clear();
+ RETURN_IF_ERROR(_writer._generate_encoded_default_seq_value(
+ _tablet_schema, *_writer._opts.rowset_ctx->partial_update_info, &cur_seq_val));
+ }
+ }
+ VLOG_DEBUG << fmt::format("start pos={}, output_block.rows()={}", pos, output_block.rows());
+
+ for (int64_t rid {pos}; rid < end; rid++) {
+ auto& skip_bitmap = skip_bitmaps->at(rid);
+ bool row_has_sequence_col = (!skip_bitmap.contains(seq_col_unique_id));
+ bool have_delete_sign =
+ (!skip_bitmap.contains(delete_sign_col_unique_id) && delete_signs[rid] != 0);
+ if (!row_has_sequence_col) {
+ append_or_merge_row(output_block, block, rid, skip_bitmap, have_delete_sign);
+ } else {
+ std::string seq_val {};
+ _writer._encode_seq_column(seq_column, rid, &seq_val);
+ if (Slice {seq_val}.compare(Slice {cur_seq_val}) >= 0) {
+ append_or_merge_row(output_block, block, rid, skip_bitmap, have_delete_sign);
+ cur_seq_val = std::move(seq_val);
+ } else {
+ VLOG_DEBUG << fmt::format(
+ "skip rid={} becasue its seq value is lower than the previous", rid);
+ }
+ }
+ }
+ return Status::OK();
+};
+
+Status BlockAggregator::aggregate_for_sequence_column(
+ vectorized::Block* block, size_t num_rows,
+ const std::vector<vectorized::IOlapColumnDataAccessor*>& key_columns,
+ vectorized::IOlapColumnDataAccessor* seq_column,
+ const std::vector<RowsetSharedPtr>& specified_rowsets,
+ std::vector<std::unique_ptr<SegmentCacheHandle>>& segment_caches) {
+ DCHECK_EQ(block->columns(), _tablet_schema.num_columns());
+ VLOG_DEBUG << fmt::format("BlockAggregator::aggregate_for_sequence_column enter: block:{}\n",
+ block->dump_data());
+ // the process logic here is the same as MemTable::_aggregate_for_flexible_partial_update_without_seq_col()
+ // after this function, there will be at most 2 rows for a specified key
+ std::vector<BitmapValue>* skip_bitmaps =
+ &(assert_cast<vectorized::ColumnBitmap*>(
+ block->get_by_position(_tablet_schema.skip_bitmap_col_idx())
+ .column->assume_mutable()
+ .get())
+ ->get_data());
+ const auto* delete_signs = BaseTablet::get_delete_sign_column_data(*block, num_rows);
+
+ auto filtered_block = _tablet_schema.create_block();
+ vectorized::MutableBlock output_block =
+ vectorized::MutableBlock::build_mutable_block(&filtered_block);
+
+ int same_key_rows {0};
+ std::string previous_key {};
+ for (size_t block_pos {0}; block_pos < num_rows; block_pos++) {
+ std::string key = _writer._full_encode_keys(key_columns, block_pos);
+ if (block_pos > 0 && previous_key == key) {
+ same_key_rows++;
+ } else {
+ if (same_key_rows > 0) {
+ RETURN_IF_ERROR(aggregate_rows(output_block, block, block_pos - same_key_rows,
+ block_pos, std::move(previous_key), skip_bitmaps,
+ delete_signs, seq_column, specified_rowsets,
+ segment_caches));
+ }
+ same_key_rows = 1;
+ }
+ previous_key = std::move(key);
+ }
+ if (same_key_rows > 0) {
+ RETURN_IF_ERROR(aggregate_rows(output_block, block, num_rows - same_key_rows, num_rows,
+ std::move(previous_key), skip_bitmaps, delete_signs,
+ seq_column, specified_rowsets, segment_caches));
+ }
+
+ block->swap(output_block.to_block());
+ VLOG_DEBUG << fmt::format(
+ "BlockAggregator::aggregate_for_sequence_column after: data.block:{}\n",
+ block->dump_data());
+ return Status::OK();
+}
+
+Status BlockAggregator::fill_sequence_column(vectorized::Block* block, size_t num_rows,
+ const FixedReadPlan& read_plan,
+ std::vector<BitmapValue>& skip_bitmaps) {
+ DCHECK(_tablet_schema.has_sequence_col());
+ std::vector<uint32_t> cids {static_cast<uint32_t>(_tablet_schema.sequence_col_idx())};
+ auto seq_col_unique_id = _tablet_schema.column(_tablet_schema.sequence_col_idx()).unique_id();
+
+ auto seq_col_block = _tablet_schema.create_block_by_cids(cids);
+ auto tmp_block = _tablet_schema.create_block_by_cids(cids);
+ std::map<uint32_t, uint32_t> read_index;
+ RETURN_IF_ERROR(read_plan.read_columns_by_plan(_tablet_schema, cids, _writer._rsid_to_rowset,
+ seq_col_block, &read_index));
+
+ auto new_seq_col_ptr = tmp_block.get_by_position(0).column->assume_mutable();
+ const auto& old_seq_col_ptr = *seq_col_block.get_by_position(0).column;
+ const auto& cur_seq_col_ptr = *block->get_by_position(_tablet_schema.sequence_col_idx()).column;
+ for (size_t block_pos {0}; block_pos < num_rows; block_pos++) {
+ if (read_index.contains(block_pos)) {
+ new_seq_col_ptr->insert_from(old_seq_col_ptr, read_index[block_pos]);
+ skip_bitmaps[block_pos].remove(seq_col_unique_id);
+ } else {
+ new_seq_col_ptr->insert_from(cur_seq_col_ptr, block_pos);
+ }
+ }
+ block->replace_by_position(_tablet_schema.sequence_col_idx(), std::move(new_seq_col_ptr));
+ return Status::OK();
+}
+
+Status BlockAggregator::aggregate_for_insert_after_delete(
Review Comment:
warning: function 'aggregate_for_insert_after_delete' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
Status BlockAggregator::aggregate_for_insert_after_delete(
^
```
<details>
<summary>Additional context</summary>
**be/src/olap/partial_update_info.cpp:987:** 96 lines including whitespace and comments (threshold 80)
```cpp
Status BlockAggregator::aggregate_for_insert_after_delete(
^
```
</details>
##########
be/src/olap/partial_update_info.cpp:
##########
@@ -652,10 +739,440 @@ Status FlexibleReadPlan::fill_non_primary_key_columns_for_row_store(
*old_value_block.get_by_position(i).column,
*block->get_by_position(cid).column, block_pos,
skip_bitmaps->at(block_pos).contains(col_uid),
+ tablet_schema.has_sequence_col()
+ ? !skip_bitmaps->at(block_pos).contains(seq_col_unique_id)
+ : false,
use_default_or_null_flag[idx], delete_sign_column_data, pos_in_old_block);
}
}
return Status::OK();
}
+BlockAggregator::BlockAggregator(segment_v2::VerticalSegmentWriter& vertical_segment_writer)
+ : _writer(vertical_segment_writer), _tablet_schema(*_writer._tablet_schema) {}
+
+void BlockAggregator::merge_one_row(vectorized::MutableBlock& dst_block,
+ vectorized::Block* src_block, int64_t rid,
+ BitmapValue& skip_bitmap) {
+ for (size_t cid {_tablet_schema.num_key_columns()}; cid < _tablet_schema.num_columns(); cid++) {
+ if (cid == _tablet_schema.skip_bitmap_col_idx()) {
+ auto& cur_skip_bitmap =
+ assert_cast<vectorized::ColumnBitmap*>(dst_block.mutable_columns()[cid].get())
+ ->get_data()
+ .back();
+ const auto& new_row_skip_bitmap =
+ assert_cast<vectorized::ColumnBitmap*>(
+ src_block->get_by_position(cid).column->assume_mutable().get())
+ ->get_data()[rid];
+ cur_skip_bitmap &= new_row_skip_bitmap;
+ continue;
+ }
+ if (!skip_bitmap.contains(_tablet_schema.column(cid).unique_id())) {
+ dst_block.mutable_columns()[cid]->pop_back(1);
+ dst_block.mutable_columns()[cid]->insert_from(*src_block->get_by_position(cid).column,
+ rid);
+ }
+ }
+ VLOG_DEBUG << fmt::format("merge a row, after merge, output_block.rows()={}, state: {}",
+ dst_block.rows(), _state.to_string());
+}
+
+void BlockAggregator::append_one_row(vectorized::MutableBlock& dst_block,
+ vectorized::Block* src_block, int64_t rid) {
+ dst_block.add_row(src_block, rid);
+ _state.rows++;
+ VLOG_DEBUG << fmt::format("append a new row, after append, output_block.rows()={}, state: {}",
+ dst_block.rows(), _state.to_string());
+}
+
+void BlockAggregator::remove_last_n_rows(vectorized::MutableBlock& dst_block, int n) {
+ if (n > 0) {
+ for (size_t cid {0}; cid < _tablet_schema.num_columns(); cid++) {
+ DCHECK_GE(dst_block.mutable_columns()[cid]->size(), n);
+ dst_block.mutable_columns()[cid]->pop_back(n);
+ }
+ }
+}
+
+void BlockAggregator::append_or_merge_row(vectorized::MutableBlock& dst_block,
+ vectorized::Block* src_block, int64_t rid,
+ BitmapValue& skip_bitmap, bool have_delete_sign) {
+ if (have_delete_sign) {
+ // remove all the previous batched rows
+ remove_last_n_rows(dst_block, _state.rows);
+ _state.rows = 0;
+ _state.has_row_with_delete_sign = true;
+
+ append_one_row(dst_block, src_block, rid);
+ } else {
+ if (_state.should_merge()) {
+ merge_one_row(dst_block, src_block, rid, skip_bitmap);
+ } else {
+ append_one_row(dst_block, src_block, rid);
+ }
+ }
+};
+
+Status BlockAggregator::aggregate_rows(
Review Comment:
warning: function 'aggregate_rows' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
Status BlockAggregator::aggregate_rows(
^
```
<details>
<summary>Additional context</summary>
**be/src/olap/partial_update_info.cpp:815:** 84 lines including whitespace and comments (threshold 80)
```cpp
Status BlockAggregator::aggregate_rows(
^
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2595452241
<details>
<summary>TPC-DS: <b>Total hot run time: 187654 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 21ec0541fbd8c9eab4a761c02f69ddb72c4856a6, data reload: false
query1 1000 389 384 384
query2 6520 2040 2048 2040
query3 6795 212 213 212
query4 33399 23280 23469 23280
query5 4373 632 468 468
query6 293 197 187 187
query7 4603 508 319 319
query8 301 236 239 236
query9 9326 2586 2580 2580
query10 479 305 262 262
query11 17680 15296 14891 14891
query12 157 104 109 104
query13 1664 526 414 414
query14 10319 6528 7102 6528
query15 238 198 175 175
query16 8075 640 449 449
query17 1590 725 542 542
query18 2124 394 305 305
query19 209 187 155 155
query20 111 108 109 108
query21 204 120 100 100
query22 4248 4273 4371 4273
query23 34127 32921 32961 32921
query24 6559 2182 2226 2182
query25 509 460 385 385
query26 1200 267 161 161
query27 2025 463 335 335
query28 5268 2472 2415 2415
query29 715 545 417 417
query30 229 184 155 155
query31 953 858 791 791
query32 96 64 62 62
query33 550 372 304 304
query34 749 862 486 486
query35 806 821 753 753
query36 1001 1018 965 965
query37 126 104 84 84
query38 4095 4228 3989 3989
query39 1483 1435 1425 1425
query40 205 111 102 102
query41 53 53 52 52
query42 133 104 101 101
query43 532 531 492 492
query44 1270 850 796 796
query45 181 172 177 172
query46 852 1037 647 647
query47 1811 1837 1802 1802
query48 378 394 311 311
query49 782 483 394 394
query50 641 648 394 394
query51 6832 6868 6885 6868
query52 101 103 90 90
query53 221 248 186 186
query54 472 480 422 422
query55 79 79 82 79
query56 269 256 243 243
query57 1176 1136 1088 1088
query58 242 242 253 242
query59 2988 3135 3120 3120
query60 293 286 308 286
query61 126 118 120 118
query62 801 717 651 651
query63 214 193 187 187
query64 4523 1052 653 653
query65 3279 3140 3168 3140
query66 1085 416 301 301
query67 16036 15564 15644 15564
query68 2174 816 554 554
query69 418 304 261 261
query70 1204 1179 1157 1157
query71 312 283 262 262
query72 6040 3859 3854 3854
query73 631 746 355 355
query74 9684 9254 8752 8752
query75 3130 3186 2695 2695
query76 2205 1183 777 777
query77 349 362 282 282
query78 10084 10252 9351 9351
query79 988 868 614 614
query80 1175 554 437 437
query81 521 281 237 237
query82 357 153 127 127
query83 176 176 151 151
query84 239 92 73 73
query85 765 350 294 294
query86 335 327 316 316
query87 4530 4418 4424 4418
query88 3663 2137 2109 2109
query89 399 331 286 286
query90 1830 193 188 188
query91 130 139 109 109
query92 57 59 56 56
query93 977 850 544 544
query94 691 398 307 307
query95 354 271 266 266
query96 484 613 278 278
query97 2801 2879 2741 2741
query98 221 203 193 193
query99 1284 1416 1267 1267
Total cold run time: 277982 ms
Total hot run time: 187654 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2609752790
TeamCity be ut coverage result:
Function Coverage: 41.92% (10938/26093)
Line Coverage: 32.17% (92321/286969)
Region Coverage: 31.29% (47280/151114)
Branch Coverage: 27.34% (23931/87540)
Coverage Report: http://coverage.selectdb-in.cc/coverage/01f1538e375d5d84cf26481834e2500342d61174_01f1538e375d5d84cf26481834e2500342d61174/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2644559374
TeamCity cloud ut coverage result:
Function Coverage: 82.38% (1061/1288)
Line Coverage: 65.85% (17540/26635)
Region Coverage: 65.38% (8654/13237)
Branch Coverage: 55.24% (4663/8442)
Coverage Report: http://coverage.selectdb-in.cc/coverage/8779c6557723172b587f0c4218e70793a3530092_8779c6557723172b587f0c4218e70793a3530092_cloud/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2644727423
<details>
<summary>ClickBench: <b>Total hot run time: 30.06 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8779c6557723172b587f0c4218e70793a3530092, data reload: false
query1 0.03 0.03 0.03
query2 0.07 0.03 0.03
query3 0.25 0.07 0.07
query4 1.61 0.10 0.11
query5 0.41 0.41 0.41
query6 1.19 0.67 0.66
query7 0.03 0.02 0.01
query8 0.04 0.03 0.03
query9 0.59 0.53 0.53
query10 0.57 0.58 0.57
query11 0.15 0.10 0.10
query12 0.13 0.11 0.10
query13 0.62 0.59 0.60
query14 2.67 2.71 2.71
query15 0.92 0.86 0.85
query16 0.38 0.38 0.37
query17 1.05 1.04 1.03
query18 0.20 0.19 0.19
query19 1.87 1.81 2.00
query20 0.02 0.01 0.02
query21 15.38 0.86 0.54
query22 0.76 1.08 0.63
query23 15.06 1.36 0.63
query24 7.53 1.19 0.34
query25 0.47 0.14 0.15
query26 0.74 0.17 0.13
query27 0.05 0.05 0.04
query28 9.06 0.93 0.42
query29 12.56 4.00 3.28
query30 0.24 0.08 0.06
query31 2.83 0.61 0.38
query32 3.22 0.55 0.47
query33 2.99 2.99 3.15
query34 15.62 5.12 4.48
query35 4.52 4.55 4.50
query36 0.70 0.50 0.49
query37 0.09 0.07 0.07
query38 0.05 0.04 0.04
query39 0.03 0.03 0.02
query40 0.16 0.13 0.14
query41 0.08 0.03 0.03
query42 0.03 0.02 0.03
query43 0.04 0.03 0.03
Total cold run time: 105.01 s
Total hot run time: 30.06 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix incorrect result when there are insert after delete for same key in one load [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #41701:
URL: https://github.com/apache/doris/pull/41701#discussion_r1796852689
##########
be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:
##########
@@ -558,6 +567,32 @@ Status VerticalSegmentWriter::_append_block_with_partial_content(RowsInBlock& da
return Status::OK();
}
+Status VerticalSegmentWriter::_filter_block_for_flexible_partial_update(
+ RowsInBlock& data, vectorized::MutableColumnPtr filter_column, int duplicate_rows,
+ std::string col_name) {
+ auto num_cols = data.block->columns();
+ auto* block = const_cast<vectorized::Block*>(data.block);
+ block->insert(
+ {std::move(filter_column), std::make_shared<vectorized::DataTypeUInt8>(), col_name});
+ RETURN_IF_ERROR(vectorized::Block::filter_block(block, num_cols, num_cols));
+ DCHECK_EQ(num_cols, data.block->columns());
+ int merged_rows = data.num_rows - block->rows();
+ VLOG_DEBUG << fmt::format(
+ "filter_block_for_flexible_partial_update[{}] after filter: "
+ "data.block:{}\n",
+ col_name, data.block->dump_data());
+ if (duplicate_rows != merged_rows) {
+ auto msg = fmt::format(
+ "filter_block_for_flexible_partial_update {}: duplicate_rows != merged_rows, "
+ "duplicate_keys={}, merged_rows={}, num_rows={}, mutable_block->rows()={}",
+ col_name, duplicate_rows, merged_rows, data.num_rows, block->rows());
+ DCHECK(false) << msg;
+ return Status::InternalError<false>(msg);
+ }
+ data.num_rows = block->rows();
+ return Status::OK();
+}
+
Status VerticalSegmentWriter::_append_block_with_flexible_partial_content(
Review Comment:
warning: function '_append_block_with_flexible_partial_content' has cognitive complexity of 86 (threshold 50) [readability-function-cognitive-complexity]
```cpp
Status VerticalSegmentWriter::_append_block_with_flexible_partial_content(
^
```
<details>
<summary>Additional context</summary>
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:613:** nesting level increased to 1
```cpp
auto get_skip_bitmaps = [&skip_bitmap_col_idx](const vectorized::Block* block) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:631:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
DBUG_EXECUTE_IF("VerticalSegmentWriter._append_block_with_flexible_partial_content.sleep",
^
```
**be/src/util/debug_points.h:36:** expanded from macro 'DBUG_EXECUTE_IF'
```cpp
if (UNLIKELY(config::enable_debug_points)) { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:631:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
DBUG_EXECUTE_IF("VerticalSegmentWriter._append_block_with_flexible_partial_content.sleep",
^
```
**be/src/util/debug_points.h:38:** expanded from macro 'DBUG_EXECUTE_IF'
```cpp
if (dp) { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:640:** nesting level increased to 1
```cpp
[&full_block, &data,
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:643:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
for (std::size_t cid {0}; cid < _num_sort_key_columns; cid++) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:645:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(_olap_data_convertor->set_source_content_with_specifid_column(
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:645:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
RETURN_IF_ERROR(_olap_data_convertor->set_source_content_with_specifid_column(
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:648:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (!status.ok()) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:656:** nesting level increased to 1
```cpp
auto encode_seq_column = [&data, &schema_has_sequence_col,
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:659:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (schema_has_sequence_col) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:661:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(_olap_data_convertor->set_source_content_with_specifid_column(
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:661:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
RETURN_IF_ERROR(_olap_data_convertor->set_source_content_with_specifid_column(
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:665:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (!status.ok()) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:676:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
RETURN_IF_ERROR(encode_key_columns(key_columns));
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:676:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(encode_key_columns(key_columns));
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:683:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
RETURN_IF_ERROR(encode_seq_column(seq_column));
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:683:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(encode_seq_column(seq_column));
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:690:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
if (schema_has_sequence_col) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:691:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(_merge_rows_for_sequence_column(data, skip_bitmaps, key_columns, seq_column,
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:691:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(_merge_rows_for_sequence_column(data, skip_bitmaps, key_columns, seq_column,
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:694:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (origin_rows != data.num_rows) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:697:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(encode_key_columns(key_columns));
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:697:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
RETURN_IF_ERROR(encode_key_columns(key_columns));
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:698:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(encode_seq_column(seq_column));
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:698:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
RETURN_IF_ERROR(encode_seq_column(seq_column));
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:707:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
RETURN_IF_ERROR(_merge_rows_for_insert_after_delete(data, skip_bitmaps, key_columns, seq_column,
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:707:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(_merge_rows_for_insert_after_delete(data, skip_bitmaps, key_columns, seq_column,
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:710:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
if (data.num_rows != origin_rows) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:713:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(encode_key_columns(key_columns));
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:713:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(encode_key_columns(key_columns));
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:714:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(encode_seq_column(seq_column));
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:714:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(encode_seq_column(seq_column));
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:721:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
for (std::size_t cid {0}; cid < _num_sort_key_columns; cid++) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:724:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(_column_writers[cid]->append(column->get_nullmap(), column->get_data(),
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:724:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(_column_writers[cid]->append(column->get_nullmap(), column->get_data(),
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:732:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
RETURN_IF_ERROR(_generate_flexible_read_plan(
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:732:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(_generate_flexible_read_plan(
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:738:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
if (config::enable_merge_on_write_correctness_check) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:744:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
RETURN_IF_ERROR(read_plan.fill_non_primary_key_columns(
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:744:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(read_plan.fill_non_primary_key_columns(
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:778:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
if (_num_rows_written != data.row_pos ||
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:787:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
RETURN_IF_ERROR(_generate_primary_key_index(_key_coders, key_columns, seq_column, data.num_rows,
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:787:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(_generate_primary_key_index(_key_coders, key_columns, seq_column, data.num_rows,
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
</details>
##########
be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:
##########
@@ -831,15 +885,18 @@
Status VerticalSegmentWriter::_merge_rows_for_sequence_column(
Review Comment:
warning: function '_merge_rows_for_sequence_column' has cognitive complexity of 95 (threshold 50) [readability-function-cognitive-complexity]
```cpp
Status VerticalSegmentWriter::_merge_rows_for_sequence_column(
^
```
<details>
<summary>Additional context</summary>
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:905:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
RETURN_IF_ERROR(_generate_encoded_default_seq_value(
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:905:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(_generate_encoded_default_seq_value(
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:915:** nesting level increased to 1
```cpp
auto get_idx = [](bool with_seq_col, bool has_delete_sign) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:916:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (!with_seq_col) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:917:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (has_delete_sign) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:919:** +1, nesting level increased to 3
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:922:** +1, nesting level increased to 2
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:923:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (has_delete_sign) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:925:** +1, nesting level increased to 3
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:930:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
for (size_t block_pos = data.row_pos; block_pos < data.row_pos + data.num_rows; block_pos++) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:936:** +1
```cpp
(!skip_bitmap.contains(delete_sign_col_unique_id) && delete_signs[block_pos] != 0);
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:937:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (delta_pos > 0 && previous_key == key) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:937:** +1
```cpp
if (delta_pos > 0 && previous_key == key) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:938:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (!has_same_rows) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:944:** +1, nesting level increased to 2
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:945:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (has_same_rows) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:948:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
if (has_row_with_seq_col && has_row_without_seq_col) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:948:** +1
```cpp
if (has_row_with_seq_col && has_row_without_seq_col) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:959:** +1
```cpp
DCHECK(st.is<KEY_NOT_FOUND>() || st.ok());
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:961:** +5, including nesting penalty of 4, nesting level increased to 5
```cpp
if (st.is<KEY_NOT_FOUND>()) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:963:** +1, nesting level increased to 5
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:968:** +5, including nesting penalty of 4, nesting level increased to 5
```cpp
if (batched_rows[2] != -1) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:972:** +5, including nesting penalty of 4, nesting level increased to 5
```cpp
if (batched_rows[3] != -1) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:977:** nesting level increased to 5
```cpp
auto remove_rows_without_seq = [&]() {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:978:** +6, including nesting penalty of 5, nesting level increased to 6
```cpp
if (batched_rows[0] != -1) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:982:** +6, including nesting penalty of 5, nesting level increased to 6
```cpp
if (batched_rows[1] != -1) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:989:** +5, including nesting penalty of 4, nesting level increased to 5
```cpp
if (batched_rows[2] != -1 && batched_rows[3] != -1) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:990:** +6, including nesting penalty of 5, nesting level increased to 6
```cpp
if (previous_seq_slice.compare(
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:993:** +1, nesting level increased to 6
```cpp
} else if (previous_seq_slice.compare(Slice {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:998:** +1, nesting level increased to 6
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1004:** +1, nesting level increased to 5
```cpp
} else if (batched_rows[2] != -1) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1005:** +6, including nesting penalty of 5, nesting level increased to 6
```cpp
if (previous_seq_slice.compare(
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1008:** +1, nesting level increased to 6
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1012:** +1, nesting level increased to 5
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1013:** +6, including nesting penalty of 5, nesting level increased to 6
```cpp
if (previous_seq_slice.compare(
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1016:** +1, nesting level increased to 6
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1030:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
if (duplicate_keys > 0) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1031:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(_filter_block_for_flexible_partial_update(
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1031:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(_filter_block_for_flexible_partial_update(
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix incorrect result when there are insert after delete for same key in one load [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2425546244
<details>
<summary>ClickBench: <b>Total hot run time: 32.74 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ae261fd99369ad3a45789fe25ed021eca52ebba7, data reload: false
query1 0.03 0.03 0.03
query2 0.07 0.03 0.03
query3 0.24 0.06 0.06
query4 1.65 0.10 0.10
query5 0.54 0.51 0.50
query6 1.13 0.73 0.73
query7 0.02 0.02 0.02
query8 0.04 0.04 0.03
query9 0.56 0.48 0.48
query10 0.56 0.53 0.54
query11 0.14 0.10 0.11
query12 0.14 0.11 0.10
query13 0.61 0.59 0.60
query14 2.72 2.76 2.76
query15 0.90 0.82 0.82
query16 0.39 0.39 0.38
query17 1.03 1.05 1.07
query18 0.19 0.20 0.20
query19 1.93 1.90 1.98
query20 0.01 0.01 0.01
query21 15.36 0.58 0.57
query22 2.33 2.93 2.45
query23 16.94 1.07 0.75
query24 2.96 0.48 2.19
query25 0.29 0.06 0.15
query26 0.57 0.15 0.14
query27 0.04 0.05 0.04
query28 10.27 1.10 1.07
query29 12.56 3.24 3.17
query30 0.25 0.06 0.07
query31 2.87 0.39 0.37
query32 3.28 0.47 0.45
query33 3.05 3.00 3.09
query34 16.83 4.49 4.49
query35 4.53 4.50 4.45
query36 0.66 0.48 0.48
query37 0.08 0.05 0.06
query38 0.04 0.04 0.04
query39 0.04 0.02 0.02
query40 0.16 0.12 0.12
query41 0.08 0.02 0.02
query42 0.03 0.02 0.02
query43 0.04 0.03 0.02
Total cold run time: 106.16 s
Total hot run time: 32.74 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2467246241
TeamCity be ut coverage result:
Function Coverage: 37.89% (9868/26047)
Line Coverage: 29.05% (82264/283149)
Region Coverage: 28.20% (42347/150156)
Branch Coverage: 24.75% (21443/86638)
Coverage Report: http://coverage.selectdb-in.cc/coverage/b24c64a101de7130f3568ae00475aa4262f94492_b24c64a101de7130f3568ae00475aa4262f94492/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix incorrect result when there are insert after delete for same key in one load [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2451451777
TeamCity be ut coverage result:
Function Coverage: 37.80% (9822/25986)
Line Coverage: 28.97% (81629/281729)
Region Coverage: 28.24% (42149/149244)
Branch Coverage: 24.82% (21384/86142)
Coverage Report: http://coverage.selectdb-in.cc/coverage/ffefc7377f05b2b19319afb73bc7375cd0370491_ffefc7377f05b2b19319afb73bc7375cd0370491/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix incorrect result when there are insert after delete for same key in one load [doris]
Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2451306961
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2490466309
<details>
<summary>TPC-DS: <b>Total hot run time: 198249 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2d818f829608fbd86f1492c0f3ea5cf6307e7494, data reload: false
query1 1238 941 932 932
query2 6226 2154 2035 2035
query3 10794 4060 4122 4060
query4 67524 32480 24061 24061
query5 4887 482 463 463
query6 322 189 183 183
query7 4946 307 294 294
query8 294 230 233 230
query9 6727 2751 2713 2713
query10 419 271 253 253
query11 15779 15273 15997 15273
query12 156 98 100 98
query13 1032 427 434 427
query14 13751 7149 7184 7149
query15 209 190 201 190
query16 7034 470 440 440
query17 1362 580 604 580
query18 1110 307 312 307
query19 201 172 166 166
query20 128 120 113 113
query21 219 102 105 102
query22 4646 4557 4706 4557
query23 34922 34689 34682 34682
query24 5660 2552 2543 2543
query25 493 405 414 405
query26 635 152 157 152
query27 1880 285 289 285
query28 4269 2518 2502 2502
query29 679 425 413 413
query30 228 160 150 150
query31 1013 823 875 823
query32 64 57 67 57
query33 447 294 287 287
query34 939 539 554 539
query35 870 744 744 744
query36 1090 977 977 977
query37 126 74 81 74
query38 4489 4303 4535 4303
query39 1543 1489 1465 1465
query40 205 99 97 97
query41 43 43 44 43
query42 107 98 100 98
query43 520 499 514 499
query44 1238 866 854 854
query45 194 178 177 177
query46 1200 739 710 710
query47 2004 1941 1909 1909
query48 437 342 342 342
query49 766 405 401 401
query50 849 401 415 401
query51 7373 7208 7134 7134
query52 102 87 92 87
query53 264 191 182 182
query54 529 401 394 394
query55 78 73 78 73
query56 249 272 247 247
query57 1285 1147 1160 1147
query58 227 224 219 219
query59 3286 2987 3046 2987
query60 270 251 246 246
query61 116 111 109 109
query62 787 676 682 676
query63 209 201 193 193
query64 1352 676 652 652
query65 3330 3194 3211 3194
query66 631 302 291 291
query67 16124 15598 15809 15598
query68 3746 580 594 580
query69 434 260 274 260
query70 1230 1093 1131 1093
query71 372 259 264 259
query72 6685 4058 4017 4017
query73 776 372 378 372
query74 10077 9026 9113 9026
query75 3421 2716 2762 2716
query76 1561 1105 1091 1091
query77 467 274 297 274
query78 10597 9447 9448 9447
query79 1903 606 605 605
query80 1139 427 432 427
query81 522 241 237 237
query82 300 107 118 107
query83 181 152 154 152
query84 282 72 68 68
query85 1079 362 303 303
query86 413 301 288 288
query87 4729 4736 4588 4588
query88 4057 2253 2225 2225
query89 418 296 296 296
query90 2017 191 196 191
query91 136 103 103 103
query92 60 50 49 49
query93 2513 548 557 548
query94 885 263 287 263
query95 346 251 258 251
query96 621 284 288 284
query97 2872 2733 2653 2653
query98 225 202 194 194
query99 1597 1316 1295 1295
Total cold run time: 317787 ms
Total hot run time: 198249 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2490453869
TeamCity be ut coverage result:
Function Coverage: 38.01% (9903/26052)
Line Coverage: 29.19% (82854/283881)
Region Coverage: 28.31% (42548/150300)
Branch Coverage: 24.87% (21569/86736)
Coverage Report: http://coverage.selectdb-in.cc/coverage/2d818f829608fbd86f1492c0f3ea5cf6307e7494_2d818f829608fbd86f1492c0f3ea5cf6307e7494/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2561937527
<details>
<summary>ClickBench: <b>Total hot run time: 30.77 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9cc0eab95240250a54bda6aa63dfd1cf1932c83e, data reload: false
query1 0.04 0.03 0.03
query2 0.07 0.03 0.03
query3 0.23 0.07 0.06
query4 1.62 0.10 0.11
query5 0.44 0.41 0.41
query6 1.13 0.64 0.65
query7 0.02 0.01 0.01
query8 0.04 0.03 0.03
query9 0.59 0.51 0.49
query10 0.54 0.58 0.55
query11 0.14 0.10 0.10
query12 0.13 0.11 0.10
query13 0.60 0.60 0.60
query14 2.72 2.69 2.74
query15 0.89 0.82 0.81
query16 0.40 0.39 0.39
query17 1.07 1.03 0.99
query18 0.22 0.21 0.20
query19 1.95 1.82 2.00
query20 0.02 0.01 0.01
query21 15.39 0.92 0.60
query22 0.76 0.95 0.69
query23 15.18 1.42 0.53
query24 3.02 0.80 1.42
query25 0.17 0.15 0.05
query26 0.33 0.14 0.12
query27 0.06 0.05 0.05
query28 14.02 1.54 1.04
query29 12.60 3.93 3.23
query30 0.24 0.09 0.07
query31 2.84 0.61 0.37
query32 3.22 0.54 0.46
query33 3.07 2.98 3.11
query34 16.80 5.14 4.45
query35 4.52 4.51 4.49
query36 0.66 0.49 0.48
query37 0.09 0.07 0.06
query38 0.05 0.04 0.04
query39 0.04 0.02 0.02
query40 0.18 0.13 0.12
query41 0.08 0.02 0.02
query42 0.04 0.03 0.02
query43 0.04 0.03 0.02
Total cold run time: 106.26 s
Total hot run time: 30.77 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2595397890
<details>
<summary>TPC-H: <b>Total hot run time: 32482 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 21ec0541fbd8c9eab4a761c02f69ddb72c4856a6, data reload: false
------ Round 1 ----------------------------------
q1 17588 5541 5446 5446
q2 2058 307 176 176
q3 10405 1271 740 740
q4 10225 966 529 529
q5 7543 2410 2236 2236
q6 197 168 137 137
q7 928 772 610 610
q8 9239 1387 1201 1201
q9 5247 4908 4915 4908
q10 6836 2342 1897 1897
q11 474 271 252 252
q12 343 356 221 221
q13 17777 3677 3059 3059
q14 241 238 209 209
q15 527 467 484 467
q16 635 614 584 584
q17 563 899 330 330
q18 6866 6436 6375 6375
q19 1211 966 555 555
q20 307 337 209 209
q21 2819 2167 2031 2031
q22 371 333 310 310
Total cold run time: 102400 ms
Total hot run time: 32482 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5554 5546 5507 5507
q2 248 335 231 231
q3 2250 2606 2312 2312
q4 1393 1875 1371 1371
q5 4370 4764 4700 4700
q6 168 159 124 124
q7 2080 1994 1832 1832
q8 2660 2851 2680 2680
q9 7334 7190 7303 7190
q10 2982 3308 2813 2813
q11 606 555 497 497
q12 671 756 628 628
q13 3492 3918 3273 3273
q14 275 295 264 264
q15 525 474 458 458
q16 678 672 652 652
q17 1259 1743 1260 1260
q18 7797 7530 7221 7221
q19 806 1223 1051 1051
q20 1988 2025 1885 1885
q21 5621 5365 4876 4876
q22 599 594 582 582
Total cold run time: 53356 ms
Total hot run time: 51407 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix incorrect result when there are insert after delete for same key in one load [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2406776414
TeamCity be ut coverage result:
Function Coverage: 37.39% (9659/25830)
Line Coverage: 28.70% (80276/279695)
Region Coverage: 28.09% (41447/147527)
Branch Coverage: 24.70% (21109/85464)
Coverage Report: http://coverage.selectdb-in.cc/coverage/8dcd7ae197fa48902b27ffb384aab71195d11193_8dcd7ae197fa48902b27ffb384aab71195d11193/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2644743232
TeamCity be ut coverage result:
Function Coverage: 42.62% (11173/26215)
Line Coverage: 32.62% (93987/288099)
Region Coverage: 31.78% (48193/151660)
Branch Coverage: 27.69% (24329/87862)
Coverage Report: http://coverage.selectdb-in.cc/coverage/8779c6557723172b587f0c4218e70793a3530092_8779c6557723172b587f0c4218e70793a3530092/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2644721930
<details>
<summary>TPC-H: <b>Total hot run time: 31424 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8779c6557723172b587f0c4218e70793a3530092, data reload: false
------ Round 1 ----------------------------------
q1 17594 5191 5078 5078
q2 2041 292 169 169
q3 10419 1310 708 708
q4 10218 1027 533 533
q5 7545 2456 2376 2376
q6 195 166 133 133
q7 923 745 614 614
q8 9307 1343 1052 1052
q9 4982 4855 4584 4584
q10 6848 2319 1887 1887
q11 459 281 262 262
q12 350 354 222 222
q13 17763 3683 3094 3094
q14 214 232 211 211
q15 510 474 474 474
q16 638 615 580 580
q17 575 887 341 341
q18 6878 6299 6160 6160
q19 1261 963 532 532
q20 315 325 191 191
q21 2897 2224 1920 1920
q22 364 318 303 303
Total cold run time: 102296 ms
Total hot run time: 31424 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5102 5112 5098 5098
q2 234 327 225 225
q3 2166 2706 2316 2316
q4 1451 1902 1414 1414
q5 4260 4139 4154 4139
q6 210 167 122 122
q7 1869 1866 1685 1685
q8 2642 2639 2599 2599
q9 7323 7120 7136 7120
q10 3032 3164 2770 2770
q11 573 520 478 478
q12 681 761 589 589
q13 3482 3880 3241 3241
q14 287 309 280 280
q15 516 464 471 464
q16 671 702 661 661
q17 1116 1605 1338 1338
q18 7452 7447 7242 7242
q19 772 815 915 815
q20 1994 2030 1902 1902
q21 5410 5120 4852 4852
q22 628 629 539 539
Total cold run time: 51871 ms
Total hot run time: 49889 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2664831995
<details>
<summary>TPC-H: <b>Total hot run time: 31519 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f94d45e93c29681112808b185d09bb4ea8965344, data reload: false
------ Round 1 ----------------------------------
q1 17606 5227 5136 5136
q2 2055 298 175 175
q3 10405 1370 726 726
q4 10294 1022 523 523
q5 7534 2430 2384 2384
q6 196 169 135 135
q7 900 757 604 604
q8 9297 1337 1070 1070
q9 4850 4571 4625 4571
q10 6837 2325 1899 1899
q11 481 280 261 261
q12 342 358 221 221
q13 17780 3664 3058 3058
q14 243 239 204 204
q15 508 472 458 458
q16 617 618 585 585
q17 578 884 339 339
q18 6720 6188 6278 6188
q19 1199 961 548 548
q20 315 327 195 195
q21 2803 2136 1926 1926
q22 367 335 313 313
Total cold run time: 101927 ms
Total hot run time: 31519 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5224 5211 5216 5211
q2 237 338 234 234
q3 2190 2650 2288 2288
q4 1422 1824 1358 1358
q5 4251 4176 4142 4142
q6 213 168 127 127
q7 1870 1849 1671 1671
q8 2688 2613 2602 2602
q9 7292 7154 7146 7146
q10 3026 3243 2784 2784
q11 574 519 501 501
q12 687 756 611 611
q13 3479 3978 3224 3224
q14 291 301 289 289
q15 531 467 466 466
q16 640 670 643 643
q17 1142 1583 1364 1364
q18 7629 7410 7278 7278
q19 774 786 858 786
q20 2015 2029 1849 1849
q21 5432 4979 5083 4979
q22 638 622 576 576
Total cold run time: 52245 ms
Total hot run time: 50129 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2635579471
run p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix incorrect result when there are insert after delete for same key in one load [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2443918025
TeamCity be ut coverage result:
Function Coverage: 37.45% (9720/25954)
Line Coverage: 28.70% (80601/280855)
Region Coverage: 28.12% (41668/148194)
Branch Coverage: 24.69% (21171/85756)
Coverage Report: http://coverage.selectdb-in.cc/coverage/d978d67bc61473d4e3aeadc3ec15eff34e8f7593_d978d67bc61473d4e3aeadc3ec15eff34e8f7593/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-3044613115
<details>
<summary>TPC-H: <b>Total hot run time: 33274 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 269e246b2133708ac5567562989edaa162f27a63, data reload: false
------ Round 1 ----------------------------------
q1 17692 5204 5047 5047
q2 1945 280 183 183
q3 10629 1338 704 704
q4 10283 1029 540 540
q5 8194 2402 2347 2347
q6 178 158 127 127
q7 894 745 599 599
q8 9301 1306 1049 1049
q9 6999 5098 5066 5066
q10 6892 2389 1949 1949
q11 471 287 285 285
q12 343 349 226 226
q13 17789 3667 3086 3086
q14 224 231 223 223
q15 539 473 481 473
q16 424 438 389 389
q17 582 860 346 346
q18 7603 7284 7224 7224
q19 1230 947 541 541
q20 325 337 221 221
q21 3908 3226 2352 2352
q22 366 341 297 297
Total cold run time: 106811 ms
Total hot run time: 33274 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5094 5146 5227 5146
q2 244 334 226 226
q3 2165 2665 2323 2323
q4 1408 1808 1346 1346
q5 4192 4507 4569 4507
q6 217 174 128 128
q7 2025 1984 1815 1815
q8 2659 2562 2625 2562
q9 7408 7250 7491 7250
q10 3113 3298 2868 2868
q11 574 518 507 507
q12 699 799 645 645
q13 3661 4012 3212 3212
q14 307 303 290 290
q15 526 493 488 488
q16 449 508 442 442
q17 1181 1652 1337 1337
q18 7856 7710 7783 7710
q19 809 793 877 793
q20 1992 2137 1925 1925
q21 5096 4589 4536 4536
q22 660 634 651 634
Total cold run time: 52335 ms
Total hot run time: 50690 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-3051763179
<details>
<summary>ClickBench: <b>Total hot run time: 30.36 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit da950276a9235965a5acc2d5ba77e2579c87ae70, data reload: false
query1 0.04 0.04 0.03
query2 0.11 0.05 0.06
query3 0.29 0.06 0.07
query4 1.60 0.08 0.08
query5 0.42 0.40 0.41
query6 1.16 0.66 0.66
query7 0.02 0.02 0.02
query8 0.06 0.05 0.05
query9 0.64 0.52 0.51
query10 0.58 0.58 0.59
query11 0.26 0.12 0.12
query12 0.26 0.14 0.13
query13 0.65 0.63 0.63
query14 0.81 0.83 0.83
query15 0.97 0.91 0.89
query16 0.39 0.38 0.37
query17 1.07 1.05 1.06
query18 0.25 0.24 0.24
query19 2.04 1.90 1.85
query20 0.01 0.01 0.01
query21 15.35 1.00 0.69
query22 0.93 1.04 0.83
query23 14.72 1.49 0.79
query24 5.21 0.62 0.30
query25 0.17 0.09 0.09
query26 0.56 0.22 0.19
query27 0.09 0.08 0.08
query28 11.00 1.22 0.60
query29 12.52 4.06 3.36
query30 0.29 0.09 0.06
query31 2.87 0.64 0.43
query32 3.24 0.61 0.53
query33 3.10 3.10 3.19
query34 16.83 5.45 4.75
query35 4.79 4.84 4.84
query36 0.65 0.51 0.52
query37 0.20 0.18 0.17
query38 0.19 0.16 0.15
query39 0.05 0.04 0.05
query40 0.20 0.16 0.16
query41 0.12 0.05 0.04
query42 0.07 0.06 0.05
query43 0.05 0.04 0.05
Total cold run time: 104.83 s
Total hot run time: 30.36 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-3056450572
<details>
<summary>ClickBench: <b>Total hot run time: 30.17 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e9db8846be973f1e4fe4bbff061ba3fe85bcf0e9, data reload: false
query1 0.04 0.04 0.03
query2 0.11 0.06 0.06
query3 0.29 0.07 0.06
query4 1.59 0.08 0.09
query5 0.42 0.42 0.41
query6 1.17 0.66 0.66
query7 0.03 0.02 0.02
query8 0.07 0.05 0.05
query9 0.65 0.52 0.52
query10 0.58 0.57 0.57
query11 0.27 0.13 0.13
query12 0.25 0.14 0.13
query13 0.64 0.62 0.63
query14 0.81 0.82 0.82
query15 0.98 0.90 0.90
query16 0.38 0.39 0.38
query17 1.06 1.05 1.07
query18 0.25 0.24 0.23
query19 2.00 1.84 1.87
query20 0.02 0.01 0.01
query21 15.36 0.96 0.68
query22 0.92 0.97 0.81
query23 14.75 1.46 0.77
query24 5.56 0.51 0.28
query25 0.17 0.09 0.09
query26 0.56 0.22 0.18
query27 0.08 0.09 0.08
query28 10.98 1.24 0.59
query29 12.54 4.10 3.32
query30 0.28 0.08 0.06
query31 2.82 0.62 0.42
query32 3.26 0.60 0.51
query33 3.13 3.08 3.22
query34 16.55 5.46 4.74
query35 4.93 4.82 4.81
query36 0.66 0.52 0.50
query37 0.21 0.18 0.18
query38 0.18 0.15 0.16
query39 0.05 0.04 0.05
query40 0.19 0.16 0.16
query41 0.11 0.06 0.06
query42 0.07 0.06 0.05
query43 0.06 0.04 0.04
Total cold run time: 105.03 s
Total hot run time: 30.17 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-3056976258
# Cloud UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/49c2fe8f4ea4a7a466b4a04119035ae5a1a9ceaf_49c2fe8f4ea4a7a466b4a04119035ae5a1a9ceaf_cloud/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/49c2fe8f4ea4a7a466b4a04119035ae5a1a9ceaf_49c2fe8f4ea4a7a466b4a04119035ae5a1a9ceaf_cloud/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 80.60% (1255/1557) |
| Line Coverage | 67.78% (21415/31596) |
| Region Coverage | 67.21% (10602/15774) |
| Branch Coverage | 56.74% (5566/9810) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-3060589737
<details>
<summary>ClickBench: <b>Total hot run time: 30.12 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8139e2d8e542a5e69903ac8e087381bed76cc5fe, data reload: false
query1 0.04 0.04 0.03
query2 0.11 0.06 0.05
query3 0.29 0.06 0.06
query4 1.60 0.08 0.08
query5 0.42 0.41 0.40
query6 1.15 0.66 0.67
query7 0.02 0.01 0.02
query8 0.06 0.05 0.04
query9 0.63 0.54 0.52
query10 0.57 0.59 0.58
query11 0.27 0.13 0.12
query12 0.25 0.13 0.14
query13 0.64 0.63 0.62
query14 0.80 0.83 0.82
query15 0.98 0.89 0.90
query16 0.39 0.38 0.39
query17 1.08 1.10 1.06
query18 0.24 0.23 0.23
query19 2.00 1.87 1.84
query20 0.02 0.02 0.01
query21 15.36 0.99 0.66
query22 0.93 1.00 0.74
query23 14.78 1.50 0.75
query24 5.44 0.56 0.29
query25 0.18 0.09 0.09
query26 0.55 0.22 0.18
query27 0.08 0.09 0.08
query28 10.97 1.19 0.59
query29 12.59 4.09 3.47
query30 0.28 0.09 0.06
query31 2.85 0.64 0.44
query32 3.24 0.61 0.51
query33 3.12 3.14 3.14
query34 16.67 5.40 4.66
query35 4.92 4.81 4.75
query36 0.66 0.53 0.51
query37 0.19 0.18 0.16
query38 0.16 0.16 0.16
query39 0.05 0.05 0.04
query40 0.19 0.16 0.16
query41 0.10 0.06 0.06
query42 0.06 0.06 0.05
query43 0.06 0.05 0.04
Total cold run time: 104.99 s
Total hot run time: 30.12 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix incorrect result when there are insert after delete for same key in one load [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #41701:
URL: https://github.com/apache/doris/pull/41701#discussion_r1801047917
##########
be/src/olap/rowset/segment_v2/segment_writer.cpp:
##########
@@ -936,45 +987,62 @@
}
Status SegmentWriter::merge_rows_for_sequence_column(
Review Comment:
warning: function 'merge_rows_for_sequence_column' has cognitive complexity of 59 (threshold 50) [readability-function-cognitive-complexity]
```cpp
Status SegmentWriter::merge_rows_for_sequence_column(
^
```
<details>
<summary>Additional context</summary>
**be/src/olap/rowset/segment_v2/segment_writer.cpp:1018:** nesting level increased to 1
```cpp
auto get_idx = [](bool with_seq_col, bool has_delete_sign) {
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:1019:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (!with_seq_col) {
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:1020:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
return (has_delete_sign ? 0 : 1);
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:1021:** +1, nesting level increased to 2
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:1022:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
return (has_delete_sign ? 2 : 3);
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:1025:** nesting level increased to 1
```cpp
auto find_rows_to_filter = [&](const std::string& key) {
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:1028:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (has_row_with_seq_col && has_row_without_seq_col) {
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:1028:** +1
```cpp
if (has_row_with_seq_col && has_row_without_seq_col) {
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:1045:** +1
```cpp
DCHECK(st.is<KEY_NOT_FOUND>() || st.ok());
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:1048:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (st.is<KEY_NOT_FOUND>()) {
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:1050:** +1, nesting level increased to 3
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:1053:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (batched_rows[2] != -1) {
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:1057:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (batched_rows[3] != -1) {
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:1062:** nesting level increased to 3
```cpp
auto remove_rows_without_seq = [&]() {
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:1063:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
if (batched_rows[0] != -1) {
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:1067:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
if (batched_rows[1] != -1) {
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:1073:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (batched_rows[2] != -1 && batched_rows[3] != -1) {
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:1075:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
if (previous_seq_slice.compare(Slice {row_with_delete_sign_encoded_seq_value}) <=
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:1078:** +1, nesting level increased to 4
```cpp
} else if (previous_seq_slice.compare(
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:1083:** +1, nesting level increased to 4
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:1089:** +1, nesting level increased to 3
```cpp
} else if (batched_rows[2] != -1) {
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:1090:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
if (previous_seq_slice.compare(Slice {row_with_delete_sign_encoded_seq_value}) <=
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:1093:** +1, nesting level increased to 4
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:1097:** +1, nesting level increased to 3
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:1098:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
if (previous_seq_slice.compare(Slice {row_without_delete_sign_encoded_seq_value}) <=
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:1101:** +1, nesting level increased to 4
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:1136:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
if (has_same_rows) {
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:1139:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
if (duplicate_keys > 0) {
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:1140:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(filter_block_for_flexible_partial_update(
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:1140:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(filter_block_for_flexible_partial_update(
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
</details>
##########
be/src/olap/rowset/segment_v2/segment_writer.cpp:
##########
@@ -695,6 +704,32 @@ Status SegmentWriter::append_block_with_partial_content(const vectorized::Block*
RETURN_IF_ERROR(append_block_with_variant_subcolumns(full_block));
return Status::OK();
}
+
+Status SegmentWriter::filter_block_for_flexible_partial_update(
+ vectorized::Block* block, size_t row_pos, size_t& num_rows,
+ vectorized::MutableColumnPtr filter_column, int duplicate_rows, std::string col_name) {
+ auto num_cols = block->columns();
+ block->insert(
+ {std::move(filter_column), std::make_shared<vectorized::DataTypeUInt8>(), col_name});
+ RETURN_IF_ERROR(vectorized::Block::filter_block(block, num_cols, num_cols));
+ DCHECK_EQ(num_cols, block->columns());
+ int merged_rows = num_rows - block->rows();
+ VLOG_DEBUG << fmt::format(
+ "filter_block_for_flexible_partial_update[{}] after filter: "
+ "data.block:{}\n",
+ col_name, block->dump_data());
+ if (duplicate_rows != merged_rows) {
+ auto msg = fmt::format(
+ "filter_block_for_flexible_partial_update {}: duplicate_rows != merged_rows, "
+ "duplicate_keys={}, merged_rows={}, num_rows={}, mutable_block->rows()={}",
+ col_name, duplicate_rows, merged_rows, num_rows, block->rows());
+ DCHECK(false) << msg;
+ return Status::InternalError<false>(msg);
+ }
+ num_rows = block->rows();
+ return Status::OK();
+}
+
Status SegmentWriter::append_block_with_flexible_partial_content(const vectorized::Block* block,
Review Comment:
warning: function 'append_block_with_flexible_partial_content' has cognitive complexity of 86 (threshold 50) [readability-function-cognitive-complexity]
```cpp
Status SegmentWriter::append_block_with_flexible_partial_content(const vectorized::Block* block,
^
```
<details>
<summary>Additional context</summary>
**be/src/olap/rowset/segment_v2/segment_writer.cpp:749:** nesting level increased to 1
```cpp
auto get_skip_bitmaps = [&skip_bitmap_col_idx](const vectorized::Block* block) {
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:766:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
DBUG_EXECUTE_IF("VerticalSegmentWriter._append_block_with_flexible_partial_content.sleep",
^
```
**be/src/util/debug_points.h:36:** expanded from macro 'DBUG_EXECUTE_IF'
```cpp
if (UNLIKELY(config::enable_debug_points)) { \
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:766:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
DBUG_EXECUTE_IF("VerticalSegmentWriter._append_block_with_flexible_partial_content.sleep",
^
```
**be/src/util/debug_points.h:38:** expanded from macro 'DBUG_EXECUTE_IF'
```cpp
if (dp) { \
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:775:** nesting level increased to 1
```cpp
[&full_block, &block, &row_pos, &num_rows,
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:778:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
for (std::size_t cid {0}; cid < _num_sort_key_columns; cid++) {
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:780:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(_olap_data_convertor->set_source_content_with_specifid_column(
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:780:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
RETURN_IF_ERROR(_olap_data_convertor->set_source_content_with_specifid_column(
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:783:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (!status.ok()) {
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:791:** nesting level increased to 1
```cpp
auto encode_seq_column = [&block, &row_pos, &num_rows,
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:794:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (_tablet_schema->has_sequence_col()) {
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:796:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(_olap_data_convertor->set_source_content_with_specifid_column(
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:796:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
RETURN_IF_ERROR(_olap_data_convertor->set_source_content_with_specifid_column(
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:799:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (!status.ok()) {
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:810:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
RETURN_IF_ERROR(encode_key_columns(key_columns));
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:810:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(encode_key_columns(key_columns));
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:817:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
RETURN_IF_ERROR(encode_seq_column(seq_column));
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:817:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(encode_seq_column(seq_column));
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:824:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
if (_tablet_schema->has_sequence_col()) {
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:825:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(merge_rows_for_sequence_column(
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:825:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(merge_rows_for_sequence_column(
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:828:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (origin_rows != num_rows) {
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:831:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(encode_key_columns(key_columns));
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:831:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
RETURN_IF_ERROR(encode_key_columns(key_columns));
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:832:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(encode_seq_column(seq_column));
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:832:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
RETURN_IF_ERROR(encode_seq_column(seq_column));
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:840:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
RETURN_IF_ERROR(_merge_rows_for_insert_after_delete(
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:840:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(_merge_rows_for_insert_after_delete(
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:843:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
if (num_rows != origin_rows) {
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:846:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(encode_key_columns(key_columns));
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:846:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(encode_key_columns(key_columns));
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:847:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(encode_seq_column(seq_column));
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:847:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(encode_seq_column(seq_column));
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:853:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
for (std::size_t cid {0}; cid < _num_sort_key_columns; cid++) {
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:856:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:856:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:864:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
RETURN_IF_ERROR(generate_flexible_read_plan(
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:864:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(generate_flexible_read_plan(
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:871:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
if (config::enable_merge_on_write_correctness_check) {
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:877:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
RETURN_IF_ERROR(read_plan.fill_non_primary_key_columns(
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:877:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(read_plan.fill_non_primary_key_columns(
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:911:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
if (_num_rows_written != row_pos ||
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:920:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
RETURN_IF_ERROR(
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:920:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix incorrect result when there are insert after delete for same key in one load [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2408544777
TeamCity be ut coverage result:
Function Coverage: 37.42% (9681/25873)
Line Coverage: 28.63% (80267/280324)
Region Coverage: 28.09% (41525/147830)
Branch Coverage: 24.67% (21110/85580)
Coverage Report: http://coverage.selectdb-in.cc/coverage/02cb8af6bdf15fd2a892f3a48fa8c8b4446761a7_02cb8af6bdf15fd2a892f3a48fa8c8b4446761a7/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix incorrect result when there are insert after delete for same key in one load [doris]
Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2408430831
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2475480756
TeamCity be ut coverage result:
Function Coverage: 37.93% (9889/26075)
Line Coverage: 29.10% (82621/283891)
Region Coverage: 28.23% (42470/150440)
Branch Coverage: 24.82% (21545/86822)
Coverage Report: http://coverage.selectdb-in.cc/coverage/8e0a62f8ba4ca7e22bf4fc6f5275459118287b41_8e0a62f8ba4ca7e22bf4fc6f5275459118287b41/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix incorrect result when there are insert after delete for same key in one load [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2422365052
<details>
<summary>TPC-DS: <b>Total hot run time: 191875 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e1f3877e946fc22d7a55d804f2346266653ae2fd, data reload: false
query1 971 378 418 378
query2 6517 2136 2100 2100
query3 6781 212 223 212
query4 34366 23597 23660 23597
query5 4422 474 463 463
query6 269 178 180 178
query7 4607 308 295 295
query8 300 241 234 234
query9 9603 2670 2696 2670
query10 494 289 284 284
query11 18035 15355 15132 15132
query12 156 110 101 101
query13 1660 437 409 409
query14 10466 7055 7931 7055
query15 329 181 194 181
query16 7374 477 442 442
query17 1750 583 582 582
query18 1946 326 309 309
query19 379 158 154 154
query20 122 108 108 108
query21 212 108 109 108
query22 4620 4352 4399 4352
query23 35118 34385 34142 34142
query24 11187 2816 2803 2803
query25 662 405 423 405
query26 1174 165 167 165
query27 2430 286 298 286
query28 7747 2437 2415 2415
query29 793 442 434 434
query30 313 163 152 152
query31 1053 809 835 809
query32 100 57 60 57
query33 788 308 294 294
query34 967 531 534 531
query35 920 744 749 744
query36 1131 935 988 935
query37 151 92 88 88
query38 4084 3950 3897 3897
query39 1474 1430 1436 1430
query40 214 99 103 99
query41 51 48 48 48
query42 121 101 100 100
query43 548 496 492 492
query44 1372 830 810 810
query45 203 168 166 166
query46 1162 704 696 696
query47 1916 1835 1793 1793
query48 465 315 328 315
query49 1172 442 430 430
query50 826 400 402 400
query51 7176 6995 6912 6912
query52 107 92 91 91
query53 265 191 181 181
query54 1258 438 429 429
query55 85 79 78 78
query56 291 284 276 276
query57 1325 1176 1133 1133
query58 256 276 248 248
query59 3202 2914 3102 2914
query60 288 271 267 267
query61 113 107 100 100
query62 886 663 672 663
query63 217 193 190 190
query64 4692 646 616 616
query65 3300 3253 3223 3223
query66 1423 327 305 305
query67 16228 15599 15673 15599
query68 4914 556 535 535
query69 444 291 288 288
query70 1219 1144 1139 1139
query71 336 286 283 283
query72 6527 4003 3989 3989
query73 809 360 353 353
query74 10107 8942 9041 8942
query75 3404 2678 2683 2678
query76 2826 1000 928 928
query77 473 297 301 297
query78 10604 9661 9606 9606
query79 1374 608 608 608
query80 2093 452 459 452
query81 595 243 242 242
query82 1224 140 141 140
query83 282 138 133 133
query84 261 72 72 72
query85 1447 295 285 285
query86 394 305 299 299
query87 4442 4321 4351 4321
query88 3412 2204 2180 2180
query89 407 294 290 290
query90 2032 189 191 189
query91 150 99 104 99
query92 71 50 50 50
query93 1084 541 535 535
query94 1165 296 301 296
query95 363 245 247 245
query96 616 276 282 276
query97 3295 3132 3136 3132
query98 229 186 196 186
query99 1639 1321 1302 1302
Total cold run time: 302871 ms
Total hot run time: 191875 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix incorrect result when there are insert after delete for same key in one load [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #41701:
URL: https://github.com/apache/doris/pull/41701#discussion_r1801059010
##########
be/src/olap/rowset/segment_v2/segment_writer.cpp:
##########
@@ -695,6 +704,32 @@ Status SegmentWriter::append_block_with_partial_content(const vectorized::Block*
RETURN_IF_ERROR(append_block_with_variant_subcolumns(full_block));
return Status::OK();
}
+
+Status SegmentWriter::filter_block_for_flexible_partial_update(
+ vectorized::Block* block, size_t row_pos, size_t& num_rows,
+ vectorized::MutableColumnPtr filter_column, int duplicate_rows, std::string col_name) {
+ auto num_cols = block->columns();
+ block->insert(
+ {std::move(filter_column), std::make_shared<vectorized::DataTypeUInt8>(), col_name});
+ RETURN_IF_ERROR(vectorized::Block::filter_block(block, num_cols, num_cols));
+ DCHECK_EQ(num_cols, block->columns());
+ int merged_rows = num_rows - block->rows();
+ VLOG_DEBUG << fmt::format(
+ "filter_block_for_flexible_partial_update[{}] after filter: "
+ "data.block:{}\n",
+ col_name, block->dump_data());
+ if (duplicate_rows != merged_rows) {
+ auto msg = fmt::format(
+ "filter_block_for_flexible_partial_update {}: duplicate_rows != merged_rows, "
+ "duplicate_keys={}, merged_rows={}, num_rows={}, mutable_block->rows()={}",
+ col_name, duplicate_rows, merged_rows, num_rows, block->rows());
+ DCHECK(false) << msg;
+ return Status::InternalError<false>(msg);
+ }
+ num_rows = block->rows();
+ return Status::OK();
+}
+
Status SegmentWriter::append_block_with_flexible_partial_content(const vectorized::Block* block,
Review Comment:
warning: function 'append_block_with_flexible_partial_content' has cognitive complexity of 86 (threshold 50) [readability-function-cognitive-complexity]
```cpp
Status SegmentWriter::append_block_with_flexible_partial_content(const vectorized::Block* block,
^
```
<details>
<summary>Additional context</summary>
**be/src/olap/rowset/segment_v2/segment_writer.cpp:749:** nesting level increased to 1
```cpp
auto get_skip_bitmaps = [&skip_bitmap_col_idx](const vectorized::Block* block) {
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:765:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
DBUG_EXECUTE_IF("VerticalSegmentWriter._append_block_with_flexible_partial_content.sleep",
^
```
**be/src/util/debug_points.h:36:** expanded from macro 'DBUG_EXECUTE_IF'
```cpp
if (UNLIKELY(config::enable_debug_points)) { \
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:765:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
DBUG_EXECUTE_IF("VerticalSegmentWriter._append_block_with_flexible_partial_content.sleep",
^
```
**be/src/util/debug_points.h:38:** expanded from macro 'DBUG_EXECUTE_IF'
```cpp
if (dp) { \
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:774:** nesting level increased to 1
```cpp
[&full_block, &block, &row_pos, &num_rows,
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:777:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
for (std::size_t cid {0}; cid < _num_sort_key_columns; cid++) {
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:779:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(_olap_data_convertor->set_source_content_with_specifid_column(
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:779:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
RETURN_IF_ERROR(_olap_data_convertor->set_source_content_with_specifid_column(
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:782:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (!status.ok()) {
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:790:** nesting level increased to 1
```cpp
auto encode_seq_column = [&block, &row_pos, &num_rows,
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:793:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (_tablet_schema->has_sequence_col()) {
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:795:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(_olap_data_convertor->set_source_content_with_specifid_column(
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:795:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
RETURN_IF_ERROR(_olap_data_convertor->set_source_content_with_specifid_column(
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:798:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (!status.ok()) {
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:809:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
RETURN_IF_ERROR(encode_key_columns(key_columns));
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:809:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(encode_key_columns(key_columns));
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:816:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
RETURN_IF_ERROR(encode_seq_column(seq_column));
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:816:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(encode_seq_column(seq_column));
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:823:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
if (_tablet_schema->has_sequence_col()) {
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:824:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(merge_rows_for_sequence_column(
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:824:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(merge_rows_for_sequence_column(
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:827:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (origin_rows != num_rows) {
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:830:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(encode_key_columns(key_columns));
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:830:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
RETURN_IF_ERROR(encode_key_columns(key_columns));
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:831:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(encode_seq_column(seq_column));
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:831:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
RETURN_IF_ERROR(encode_seq_column(seq_column));
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:839:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
RETURN_IF_ERROR(merge_rows_for_insert_after_delete(
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:839:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(merge_rows_for_insert_after_delete(
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:842:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
if (num_rows != origin_rows) {
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:845:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(encode_key_columns(key_columns));
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:845:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(encode_key_columns(key_columns));
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:846:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(encode_seq_column(seq_column));
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:846:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(encode_seq_column(seq_column));
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:852:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
for (std::size_t cid {0}; cid < _num_sort_key_columns; cid++) {
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:855:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:855:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:863:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
RETURN_IF_ERROR(generate_flexible_read_plan(
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:863:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(generate_flexible_read_plan(
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:870:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
if (config::enable_merge_on_write_correctness_check) {
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:876:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
RETURN_IF_ERROR(read_plan.fill_non_primary_key_columns(
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:876:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(read_plan.fill_non_primary_key_columns(
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:910:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
if (_num_rows_written != row_pos ||
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:919:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
RETURN_IF_ERROR(
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:919:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
</details>
##########
be/src/olap/rowset/segment_v2/segment_writer.cpp:
##########
@@ -936,45 +986,62 @@
}
Status SegmentWriter::merge_rows_for_sequence_column(
Review Comment:
warning: function 'merge_rows_for_sequence_column' has cognitive complexity of 71 (threshold 50) [readability-function-cognitive-complexity]
```cpp
Status SegmentWriter::merge_rows_for_sequence_column(
^
```
<details>
<summary>Additional context</summary>
**be/src/olap/rowset/segment_v2/segment_writer.cpp:1017:** nesting level increased to 1
```cpp
auto get_idx = [](bool with_seq_col, bool has_delete_sign) {
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:1018:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (!with_seq_col) {
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:1019:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
return (has_delete_sign ? 0 : 1);
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:1020:** +1, nesting level increased to 2
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:1021:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
return (has_delete_sign ? 2 : 3);
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:1024:** nesting level increased to 1
```cpp
auto find_rows_to_filter = [&](const std::string& key) {
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:1027:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (has_row_with_seq_col && has_row_without_seq_col) {
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:1027:** +1
```cpp
if (has_row_with_seq_col && has_row_without_seq_col) {
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:1044:** +1
```cpp
DCHECK(st.is<KEY_NOT_FOUND>() || st.ok());
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:1047:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (st.is<KEY_NOT_FOUND>()) {
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:1049:** +1, nesting level increased to 3
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:1052:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (batched_rows[2] != -1) {
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:1056:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (batched_rows[3] != -1) {
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:1061:** nesting level increased to 3
```cpp
auto remove_rows_without_seq = [&]() {
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:1062:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
if (batched_rows[0] != -1) {
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:1066:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
if (batched_rows[1] != -1) {
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:1072:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (batched_rows[2] != -1 && batched_rows[3] != -1) {
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:1074:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
if (previous_seq_slice.compare(Slice {row_with_delete_sign_encoded_seq_value}) <=
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:1077:** +1, nesting level increased to 4
```cpp
} else if (previous_seq_slice.compare(
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:1082:** +1, nesting level increased to 4
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:1088:** +1, nesting level increased to 3
```cpp
} else if (batched_rows[2] != -1) {
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:1089:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
if (previous_seq_slice.compare(Slice {row_with_delete_sign_encoded_seq_value}) <=
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:1092:** +1, nesting level increased to 4
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:1096:** +1, nesting level increased to 3
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:1097:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
if (previous_seq_slice.compare(Slice {row_without_delete_sign_encoded_seq_value}) <=
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:1100:** +1, nesting level increased to 4
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:1110:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
for (size_t block_pos = row_pos; block_pos < row_pos + num_rows; block_pos++) {
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:1116:** +1
```cpp
(!skip_bitmap.contains(delete_sign_col_unique_id) && delete_signs[block_pos] != 0);
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:1119:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (delta_pos > 0 && previous_key == key) {
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:1119:** +1
```cpp
if (delta_pos > 0 && previous_key == key) {
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:1120:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (!has_same_rows) {
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:1126:** +1, nesting level increased to 2
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:1127:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (has_same_rows) {
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:1135:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
if (has_same_rows) {
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:1138:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
if (duplicate_keys > 0) {
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:1139:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(filter_block_for_flexible_partial_update(
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/segment_writer.cpp:1139:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(filter_block_for_flexible_partial_update(
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix incorrect result when there are insert after delete for same key in one load [doris]
Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2413756078
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix incorrect result when there are insert after delete for same key in one load [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #41701:
URL: https://github.com/apache/doris/pull/41701#discussion_r1825467378
##########
be/src/olap/partial_update_info.cpp:
##########
@@ -658,4 +672,431 @@ Status FlexibleReadPlan::fill_non_primary_key_columns_for_row_store(
return Status::OK();
}
+BlockAggregator::BlockAggregator(segment_v2::VerticalSegmentWriter& vertical_segment_writer)
+ : _writer(vertical_segment_writer), _tablet_schema(*_writer._tablet_schema) {}
+
+void BlockAggregator::merge_one_row(vectorized::MutableBlock& dst_block,
+ vectorized::Block* src_block, int64_t rid,
+ BitmapValue& skip_bitmap) {
+ for (size_t cid {_tablet_schema.num_key_columns()}; cid < _tablet_schema.num_columns(); cid++) {
+ if (cid == _tablet_schema.skip_bitmap_col_idx()) {
+ auto& cur_skip_bitmap =
+ assert_cast<vectorized::ColumnBitmap*>(dst_block.mutable_columns()[cid].get())
+ ->get_data()
+ .back();
+ const auto& new_row_skip_bitmap =
+ assert_cast<vectorized::ColumnBitmap*>(
+ src_block->get_by_position(cid).column->assume_mutable().get())
+ ->get_data()[rid];
+ cur_skip_bitmap &= new_row_skip_bitmap;
+ continue;
+ }
+ if (!skip_bitmap.contains(_tablet_schema.column(cid).unique_id())) {
+ dst_block.mutable_columns()[cid]->pop_back(1);
+ dst_block.mutable_columns()[cid]->insert_from(*src_block->get_by_position(cid).column,
+ rid);
+ }
+ }
+ VLOG_DEBUG << fmt::format("merge a row, after merge, output_block.rows()={}, state: {}",
+ dst_block.rows(), _state.to_string());
+}
+
+void BlockAggregator::append_one_row(vectorized::MutableBlock& dst_block,
+ vectorized::Block* src_block, int64_t rid) {
+ dst_block.add_row(src_block, rid);
+ _state.rows++;
+ VLOG_DEBUG << fmt::format("append a new row, after append, output_block.rows()={}, state: {}",
+ dst_block.rows(), _state.to_string());
+}
+
+void BlockAggregator::remove_last_n_rows(vectorized::MutableBlock& dst_block, int n) {
+ if (n > 0) {
+ for (size_t cid {0}; cid < _tablet_schema.num_columns(); cid++) {
+ DCHECK_GE(dst_block.mutable_columns()[cid]->size(), n);
+ dst_block.mutable_columns()[cid]->pop_back(n);
+ }
+ }
+}
+
+void BlockAggregator::append_or_merge_row(vectorized::MutableBlock& dst_block,
+ vectorized::Block* src_block, int64_t rid,
+ BitmapValue& skip_bitmap, bool have_delete_sign) {
+ if (have_delete_sign) {
+ // remove all the previous batched rows
+ remove_last_n_rows(dst_block, _state.rows);
+ _state.rows = 0;
+ _state.has_row_with_delete_sign = true;
+
+ append_one_row(dst_block, src_block, rid);
+ } else {
+ if (_state.should_merge()) {
+ merge_one_row(dst_block, src_block, rid, skip_bitmap);
+ } else {
+ append_one_row(dst_block, src_block, rid);
+ }
+ }
+};
+
+Status BlockAggregator::aggregate_rows(
Review Comment:
warning: function 'aggregate_rows' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
Status BlockAggregator::aggregate_rows(
^
```
<details>
<summary>Additional context</summary>
**be/src/olap/partial_update_info.cpp:739:** 84 lines including whitespace and comments (threshold 80)
```cpp
Status BlockAggregator::aggregate_rows(
^
```
</details>
##########
be/src/olap/partial_update_info.cpp:
##########
@@ -658,4 +672,431 @@
return Status::OK();
}
+BlockAggregator::BlockAggregator(segment_v2::VerticalSegmentWriter& vertical_segment_writer)
+ : _writer(vertical_segment_writer), _tablet_schema(*_writer._tablet_schema) {}
+
+void BlockAggregator::merge_one_row(vectorized::MutableBlock& dst_block,
+ vectorized::Block* src_block, int64_t rid,
+ BitmapValue& skip_bitmap) {
+ for (size_t cid {_tablet_schema.num_key_columns()}; cid < _tablet_schema.num_columns(); cid++) {
+ if (cid == _tablet_schema.skip_bitmap_col_idx()) {
+ auto& cur_skip_bitmap =
+ assert_cast<vectorized::ColumnBitmap*>(dst_block.mutable_columns()[cid].get())
+ ->get_data()
+ .back();
+ const auto& new_row_skip_bitmap =
+ assert_cast<vectorized::ColumnBitmap*>(
+ src_block->get_by_position(cid).column->assume_mutable().get())
+ ->get_data()[rid];
+ cur_skip_bitmap &= new_row_skip_bitmap;
+ continue;
+ }
+ if (!skip_bitmap.contains(_tablet_schema.column(cid).unique_id())) {
+ dst_block.mutable_columns()[cid]->pop_back(1);
+ dst_block.mutable_columns()[cid]->insert_from(*src_block->get_by_position(cid).column,
+ rid);
+ }
+ }
+ VLOG_DEBUG << fmt::format("merge a row, after merge, output_block.rows()={}, state: {}",
+ dst_block.rows(), _state.to_string());
+}
+
+void BlockAggregator::append_one_row(vectorized::MutableBlock& dst_block,
+ vectorized::Block* src_block, int64_t rid) {
+ dst_block.add_row(src_block, rid);
+ _state.rows++;
+ VLOG_DEBUG << fmt::format("append a new row, after append, output_block.rows()={}, state: {}",
+ dst_block.rows(), _state.to_string());
+}
+
+void BlockAggregator::remove_last_n_rows(vectorized::MutableBlock& dst_block, int n) {
+ if (n > 0) {
+ for (size_t cid {0}; cid < _tablet_schema.num_columns(); cid++) {
+ DCHECK_GE(dst_block.mutable_columns()[cid]->size(), n);
+ dst_block.mutable_columns()[cid]->pop_back(n);
+ }
+ }
+}
+
+void BlockAggregator::append_or_merge_row(vectorized::MutableBlock& dst_block,
+ vectorized::Block* src_block, int64_t rid,
+ BitmapValue& skip_bitmap, bool have_delete_sign) {
+ if (have_delete_sign) {
+ // remove all the previous batched rows
+ remove_last_n_rows(dst_block, _state.rows);
+ _state.rows = 0;
+ _state.has_row_with_delete_sign = true;
+
+ append_one_row(dst_block, src_block, rid);
+ } else {
+ if (_state.should_merge()) {
+ merge_one_row(dst_block, src_block, rid, skip_bitmap);
+ } else {
+ append_one_row(dst_block, src_block, rid);
+ }
+ }
+};
+
+Status BlockAggregator::aggregate_rows(
+ vectorized::MutableBlock& output_block, vectorized::Block* block, int64_t start,
+ int64_t end, std::string key, std::vector<BitmapValue>* skip_bitmaps,
+ const signed char* delete_signs, vectorized::IOlapColumnDataAccessor* seq_column,
+ const std::vector<RowsetSharedPtr>& specified_rowsets,
+ std::vector<std::unique_ptr<SegmentCacheHandle>>& segment_caches) {
+ VLOG_DEBUG << fmt::format("merge rows in range=[{}-{})", start, end);
+ if (end - start == 1) {
+ output_block.add_row(block, start);
+ VLOG_DEBUG << fmt::format("append a row directly, rid={}", start);
+ return Status::OK();
+ }
+
+ auto seq_col_unique_id = _tablet_schema.column(_tablet_schema.sequence_col_idx()).unique_id();
+ auto delete_sign_col_unique_id =
+ _tablet_schema.column(_tablet_schema.delete_sign_idx()).unique_id();
+
+ _state.reset();
+
+ RowLocation loc;
+ RowsetSharedPtr rowset;
+ std::string previous_encoded_seq_value {};
+ Status st = _writer._tablet->lookup_row_key(
+ key, &_tablet_schema, false, specified_rowsets, &loc, _writer._mow_context->max_version,
+ segment_caches, &rowset, true, &previous_encoded_seq_value);
+ int64_t pos = start;
+ bool is_expected_st = (st.is<ErrorCode::KEY_NOT_FOUND>() || st.ok());
+ DCHECK(is_expected_st || st.is<ErrorCode::MEM_LIMIT_EXCEEDED>())
+ << "[BlockAggregator::aggregate_rows] unexpected error status while lookup_row_key:"
+ << st;
+ if (!is_expected_st) {
+ return st;
+ }
+
+ std::string cur_seq_val;
+ if (st.ok()) {
+ for (pos = start; pos < end; pos++) {
+ auto& skip_bitmap = skip_bitmaps->at(pos);
+ bool row_has_sequence_col = (!skip_bitmap.contains(seq_col_unique_id));
+ // Discard all the rows whose seq value is smaller than previous_encoded_seq_value.
+ if (row_has_sequence_col) {
+ std::string seq_val {};
+ _writer._encode_seq_column(seq_column, pos, &seq_val);
+ if (Slice {seq_val}.compare(Slice {previous_encoded_seq_value}) < 0) {
+ continue;
+ }
+ cur_seq_val = std::move(seq_val);
+ break;
+ }
+ cur_seq_val = std::move(previous_encoded_seq_value);
+ break;
+ }
+ } else {
+ pos = start;
+ auto& skip_bitmap = skip_bitmaps->at(pos);
+ bool row_has_sequence_col = (!skip_bitmap.contains(seq_col_unique_id));
+ if (row_has_sequence_col) {
+ std::string seq_val {};
+ // for rows that don't specify seqeunce col, seq_val will be encoded to minial value
+ _writer._encode_seq_column(seq_column, pos, &seq_val);
+ cur_seq_val = std::move(seq_val);
+ } else {
+ cur_seq_val.clear();
+ RETURN_IF_ERROR(_writer._generate_encoded_default_seq_value(
+ _tablet_schema, *_writer._opts.rowset_ctx->partial_update_info, &cur_seq_val));
+ }
+ }
+ VLOG_DEBUG << fmt::format("start pos={}, output_block.rows()={}", pos, output_block.rows());
+
+ for (int64_t rid {pos}; rid < end; rid++) {
+ auto& skip_bitmap = skip_bitmaps->at(rid);
+ bool row_has_sequence_col = (!skip_bitmap.contains(seq_col_unique_id));
+ bool have_delete_sign =
+ (!skip_bitmap.contains(delete_sign_col_unique_id) && delete_signs[rid] != 0);
+ if (!row_has_sequence_col) {
+ append_or_merge_row(output_block, block, rid, skip_bitmap, have_delete_sign);
+ } else {
+ std::string seq_val {};
+ _writer._encode_seq_column(seq_column, rid, &seq_val);
+ if (Slice {seq_val}.compare(Slice {cur_seq_val}) >= 0) {
+ append_or_merge_row(output_block, block, rid, skip_bitmap, have_delete_sign);
+ cur_seq_val = std::move(seq_val);
+ } else {
+ VLOG_DEBUG << fmt::format(
+ "skip rid={} becasue its seq value is lower than the previous", rid);
+ }
+ }
+ }
+ return Status::OK();
+};
+
+Status BlockAggregator::aggregate_for_sequence_column(
+ vectorized::Block* block, size_t num_rows,
+ const std::vector<vectorized::IOlapColumnDataAccessor*>& key_columns,
+ vectorized::IOlapColumnDataAccessor* seq_column,
+ const std::vector<RowsetSharedPtr>& specified_rowsets,
+ std::vector<std::unique_ptr<SegmentCacheHandle>>& segment_caches) {
+ DCHECK_EQ(block->columns(), _tablet_schema.num_columns());
+ VLOG_DEBUG << fmt::format("BlockAggregator::aggregate_for_sequence_column enter: block:{}\n",
+ block->dump_data());
+ // the process logic here is the same as MemTable::_aggregate_for_flexible_partial_update_without_seq_col()
+ // after this function, there will be at most 2 rows for a specified key
+ std::vector<BitmapValue>* skip_bitmaps =
+ &(assert_cast<vectorized::ColumnBitmap*>(
+ block->get_by_position(_tablet_schema.skip_bitmap_col_idx())
+ .column->assume_mutable()
+ .get())
+ ->get_data());
+ const auto* delete_signs = BaseTablet::get_delete_sign_column_data(*block, num_rows);
+
+ auto filtered_block = _tablet_schema.create_block();
+ vectorized::MutableBlock output_block =
+ vectorized::MutableBlock::build_mutable_block(&filtered_block);
+
+ int same_key_rows {0};
+ std::string previous_key {};
+ for (size_t block_pos {0}; block_pos < num_rows; block_pos++) {
+ std::string key = _writer._full_encode_keys(key_columns, block_pos);
+ if (block_pos > 0 && previous_key == key) {
+ same_key_rows++;
+ } else {
+ if (same_key_rows > 0) {
+ RETURN_IF_ERROR(aggregate_rows(output_block, block, block_pos - same_key_rows,
+ block_pos, std::move(previous_key), skip_bitmaps,
+ delete_signs, seq_column, specified_rowsets,
+ segment_caches));
+ }
+ same_key_rows = 1;
+ }
+ previous_key = std::move(key);
+ }
+ if (same_key_rows > 0) {
+ RETURN_IF_ERROR(aggregate_rows(output_block, block, num_rows - same_key_rows, num_rows,
+ std::move(previous_key), skip_bitmaps, delete_signs,
+ seq_column, specified_rowsets, segment_caches));
+ }
+
+ block->swap(output_block.to_block());
+ VLOG_DEBUG << fmt::format(
+ "BlockAggregator::aggregate_for_sequence_column after: data.block:{}\n",
+ block->dump_data());
+ return Status::OK();
+}
+
+Status BlockAggregator::fill_sequence_column(vectorized::Block* block, size_t num_rows,
+ const FixedReadPlan& read_plan,
+ std::vector<BitmapValue>& skip_bitmaps) {
+ DCHECK(_tablet_schema.has_sequence_col());
+ std::vector<uint32_t> cids {static_cast<uint32_t>(_tablet_schema.sequence_col_idx())};
+ auto seq_col_unique_id = _tablet_schema.column(_tablet_schema.sequence_col_idx()).unique_id();
+
+ auto seq_col_block = _tablet_schema.create_block_by_cids(cids);
+ auto tmp_block = _tablet_schema.create_block_by_cids(cids);
+ std::map<uint32_t, uint32_t> read_index;
+ RETURN_IF_ERROR(read_plan.read_columns_by_plan(_tablet_schema, cids, _writer._rsid_to_rowset,
+ seq_col_block, &read_index));
+
+ auto new_seq_col_ptr = tmp_block.get_by_position(0).column->assume_mutable();
+ const auto& old_seq_col_ptr = *seq_col_block.get_by_position(0).column;
+ const auto& cur_seq_col_ptr = *block->get_by_position(_tablet_schema.sequence_col_idx()).column;
+ for (size_t block_pos {0}; block_pos < num_rows; block_pos++) {
+ if (read_index.contains(block_pos)) {
+ new_seq_col_ptr->insert_from(old_seq_col_ptr, read_index[block_pos]);
+ skip_bitmaps[block_pos].remove(seq_col_unique_id);
+ } else {
+ new_seq_col_ptr->insert_from(cur_seq_col_ptr, block_pos);
+ }
+ }
+ block->replace_by_position(_tablet_schema.sequence_col_idx(), std::move(new_seq_col_ptr));
+ return Status::OK();
+}
+
+Status BlockAggregator::aggregate_for_insert_after_delete(
Review Comment:
warning: function 'aggregate_for_insert_after_delete' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
Status BlockAggregator::aggregate_for_insert_after_delete(
^
```
<details>
<summary>Additional context</summary>
**be/src/olap/partial_update_info.cpp:911:** 96 lines including whitespace and comments (threshold 80)
```cpp
Status BlockAggregator::aggregate_for_insert_after_delete(
^
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix incorrect result when there are insert after delete for same key in one load [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2451467988
<details>
<summary>TPC-DS: <b>Total hot run time: 195661 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ffefc7377f05b2b19319afb73bc7375cd0370491, data reload: false
query1 1204 916 888 888
query2 6230 2061 2043 2043
query3 10856 4172 3985 3985
query4 67920 29749 23527 23527
query5 5095 438 444 438
query6 412 175 167 167
query7 5775 306 291 291
query8 302 212 221 212
query9 9474 2787 2760 2760
query10 452 253 247 247
query11 17537 15527 15966 15527
query12 157 108 105 105
query13 1543 438 418 418
query14 10719 7263 6763 6763
query15 201 176 188 176
query16 7069 473 409 409
query17 1045 563 558 558
query18 1837 297 294 294
query19 200 149 155 149
query20 120 109 122 109
query21 200 101 98 98
query22 4659 4439 4235 4235
query23 34221 33792 34056 33792
query24 5915 2714 2741 2714
query25 482 380 371 371
query26 652 154 152 152
query27 1687 283 290 283
query28 4027 2481 2428 2428
query29 669 399 401 399
query30 232 149 149 149
query31 999 799 817 799
query32 66 52 54 52
query33 408 257 253 253
query34 919 523 511 511
query35 857 752 726 726
query36 1062 940 946 940
query37 118 72 71 71
query38 4353 4390 4292 4292
query39 1467 1424 1383 1383
query40 190 93 95 93
query41 45 43 40 40
query42 110 93 94 93
query43 530 492 492 492
query44 1203 812 819 812
query45 183 167 164 164
query46 1116 709 689 689
query47 1964 1834 1889 1834
query48 406 316 318 316
query49 719 399 365 365
query50 802 396 392 392
query51 7333 7051 7108 7051
query52 102 86 87 86
query53 255 176 177 176
query54 510 393 385 385
query55 76 75 75 75
query56 254 264 243 243
query57 1264 1208 1166 1166
query58 208 193 193 193
query59 3332 3131 3071 3071
query60 289 235 236 235
query61 100 116 121 116
query62 796 686 671 671
query63 218 192 186 186
query64 1463 703 691 691
query65 3286 3187 3223 3187
query66 695 325 316 316
query67 16076 15793 15569 15569
query68 2963 593 589 589
query69 409 262 258 258
query70 1226 1159 1178 1159
query71 353 248 252 248
query72 5323 4192 4047 4047
query73 761 359 365 359
query74 9708 9007 9121 9007
query75 3368 2653 2696 2653
query76 1452 1040 1108 1040
query77 478 270 276 270
query78 10291 9402 9370 9370
query79 1339 592 597 592
query80 839 424 431 424
query81 505 237 240 237
query82 911 118 119 118
query83 157 144 142 142
query84 270 69 69 69
query85 875 301 278 278
query86 335 307 299 299
query87 4943 4789 4773 4773
query88 3800 2201 2172 2172
query89 407 295 297 295
query90 2004 187 176 176
query91 127 98 97 97
query92 62 47 46 46
query93 1436 567 550 550
query94 800 287 272 272
query95 342 237 243 237
query96 615 288 275 275
query97 2871 2693 2737 2693
query98 206 202 201 201
query99 1602 1306 1294 1294
Total cold run time: 315444 ms
Total hot run time: 195661 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix incorrect result when there are insert after delete for same key in one load [doris]
Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2434459707
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix incorrect result when there are insert after delete for same key in one load [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2443413917
<details>
<summary>TPC-H: <b>Total hot run time: 41731 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2bde941d7f0c31261a07d9bdc7873b8f1521d792, data reload: false
------ Round 1 ----------------------------------
q1 17570 7498 7365 7365
q2 2046 158 152 152
q3 10582 1161 1230 1161
q4 10231 803 864 803
q5 7772 3157 3103 3103
q6 237 144 143 143
q7 1057 618 592 592
q8 9346 2043 2037 2037
q9 6717 6527 6516 6516
q10 7053 2485 2437 2437
q11 452 241 247 241
q12 411 217 217 217
q13 17801 3020 3007 3007
q14 247 228 219 219
q15 547 521 515 515
q16 633 572 596 572
q17 965 600 528 528
q18 7380 6727 6676 6676
q19 1362 1099 1109 1099
q20 481 188 179 179
q21 4080 3155 3236 3155
q22 1107 1052 1014 1014
Total cold run time: 108077 ms
Total hot run time: 41731 ms
----- Round 2, with runtime_filter_mode=off -----
q1 7401 7312 7401 7312
q2 334 231 235 231
q3 2916 2825 2779 2779
q4 1952 1685 1711 1685
q5 5521 5505 5540 5505
q6 225 137 137 137
q7 2128 1712 1696 1696
q8 3272 3459 3465 3459
q9 8578 8655 8623 8623
q10 3488 3496 3494 3494
q11 585 481 486 481
q12 818 620 595 595
q13 8649 2982 2997 2982
q14 304 260 263 260
q15 552 507 502 502
q16 664 621 628 621
q17 1860 1596 1570 1570
q18 7821 7376 7528 7376
q19 1719 1552 1640 1552
q20 2098 1805 1822 1805
q21 5412 5225 5263 5225
q22 1113 1038 1032 1032
Total cold run time: 67410 ms
Total hot run time: 58922 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2664711477
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2493454589
<details>
<summary>TPC-H: <b>Total hot run time: 40631 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 61a7fd52e74446b3f27eccd533514d31d5b431bf, data reload: false
------ Round 1 ----------------------------------
q1 17671 7619 7359 7359
q2 2055 183 172 172
q3 10663 1175 1273 1175
q4 10559 771 770 770
q5 7654 2803 2847 2803
q6 252 156 151 151
q7 1017 643 614 614
q8 9256 1912 1961 1912
q9 6624 6436 6477 6436
q10 7025 2333 2359 2333
q11 454 286 266 266
q12 437 216 218 216
q13 17782 3100 3083 3083
q14 246 222 214 214
q15 583 520 517 517
q16 644 598 582 582
q17 1023 521 495 495
q18 7555 6856 6817 6817
q19 1350 1049 1056 1049
q20 479 176 177 176
q21 3973 3173 3265 3173
q22 381 318 321 318
Total cold run time: 107683 ms
Total hot run time: 40631 ms
----- Round 2, with runtime_filter_mode=off -----
q1 7262 7307 7252 7252
q2 330 233 226 226
q3 2959 2982 2991 2982
q4 2085 1851 1906 1851
q5 5740 5743 5722 5722
q6 224 145 146 145
q7 2308 1842 1859 1842
q8 3446 3600 3636 3600
q9 8970 9037 9102 9037
q10 3633 3603 3585 3585
q11 609 499 507 499
q12 825 594 662 594
q13 12382 3278 3229 3229
q14 307 288 265 265
q15 574 531 523 523
q16 691 663 655 655
q17 1903 1656 1647 1647
q18 8399 8035 7608 7608
q19 1710 1694 1645 1645
q20 2137 1867 1992 1867
q21 5639 5324 5471 5324
q22 652 568 584 568
Total cold run time: 72785 ms
Total hot run time: 60666 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2497299019
<details>
<summary>TPC-DS: <b>Total hot run time: 196943 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 996d2c977562785418ae9b0265c994d07cca4e51, data reload: false
query1 1234 952 953 952
query2 6233 2115 2117 2115
query3 10949 4249 4011 4011
query4 67692 28559 23594 23594
query5 5064 462 454 454
query6 415 195 190 190
query7 5607 314 291 291
query8 319 236 237 236
query9 8654 2645 2637 2637
query10 454 261 254 254
query11 17430 15316 16044 15316
query12 170 110 109 109
query13 1535 437 432 432
query14 10965 7847 7286 7286
query15 214 191 193 191
query16 7026 510 488 488
query17 1105 569 567 567
query18 1770 304 312 304
query19 235 151 149 149
query20 125 111 110 110
query21 208 104 104 104
query22 4774 4700 4432 4432
query23 34909 34245 34185 34185
query24 5464 2589 2522 2522
query25 496 407 395 395
query26 644 155 151 151
query27 2019 289 288 288
query28 4140 2464 2430 2430
query29 673 438 407 407
query30 215 157 147 147
query31 992 858 847 847
query32 90 54 54 54
query33 408 284 297 284
query34 938 503 519 503
query35 833 725 737 725
query36 1090 976 984 976
query37 122 74 83 74
query38 4508 4434 4501 4434
query39 1513 1465 1476 1465
query40 201 105 98 98
query41 45 40 45 40
query42 112 98 100 98
query43 552 516 502 502
query44 1176 814 807 807
query45 185 169 167 167
query46 1154 714 691 691
query47 2022 1921 1919 1919
query48 400 314 321 314
query49 721 401 380 380
query50 847 403 391 391
query51 7435 7201 7100 7100
query52 102 86 89 86
query53 259 175 178 175
query54 505 391 382 382
query55 73 72 75 72
query56 251 236 233 233
query57 1308 1174 1181 1174
query58 228 221 206 206
query59 3246 3019 2989 2989
query60 266 239 244 239
query61 113 113 110 110
query62 790 668 663 663
query63 214 197 188 188
query64 1383 679 636 636
query65 3280 3244 3220 3220
query66 696 294 304 294
query67 15928 15571 15875 15571
query68 4282 547 555 547
query69 424 255 261 255
query70 1220 1164 1149 1149
query71 360 243 244 243
query72 6466 4100 3963 3963
query73 769 353 349 349
query74 10375 9084 9103 9084
query75 3338 2673 2681 2673
query76 1963 1064 1051 1051
query77 487 274 271 271
query78 10402 9471 9427 9427
query79 2185 574 613 574
query80 1386 412 450 412
query81 531 236 233 233
query82 1223 117 120 117
query83 213 234 142 142
query84 281 69 74 69
query85 1016 307 299 299
query86 436 301 294 294
query87 4714 4597 4617 4597
query88 3743 2180 2146 2146
query89 417 300 300 300
query90 2015 188 183 183
query91 140 156 105 105
query92 70 50 50 50
query93 3023 532 550 532
query94 919 300 298 298
query95 359 247 250 247
query96 640 274 280 274
query97 2814 2726 2718 2718
query98 227 200 200 200
query99 1572 1308 1287 1287
Total cold run time: 322734 ms
Total hot run time: 196943 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2642240069
<details>
<summary>TPC-DS: <b>Total hot run time: 190482 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ebcaf82a4f4c0679d1c2fcdb77d2a5355d5a12b6, data reload: false
query1 1294 961 929 929
query2 6318 1857 1893 1857
query3 11096 4532 4578 4532
query4 54057 25095 23026 23026
query5 5282 552 484 484
query6 380 207 189 189
query7 5173 535 295 295
query8 326 266 229 229
query9 6865 2608 2600 2600
query10 433 317 257 257
query11 15301 15083 14772 14772
query12 160 110 109 109
query13 1221 536 405 405
query14 10949 6904 6506 6506
query15 203 196 176 176
query16 6994 664 528 528
query17 1070 739 547 547
query18 1484 395 311 311
query19 195 189 172 172
query20 126 130 121 121
query21 207 130 112 112
query22 4604 4655 4509 4509
query23 33998 33474 33406 33406
query24 5650 2468 2406 2406
query25 473 441 398 398
query26 663 274 156 156
query27 1673 498 339 339
query28 2881 2454 2463 2454
query29 562 543 436 436
query30 214 200 160 160
query31 892 898 840 840
query32 77 65 61 61
query33 436 358 305 305
query34 806 884 513 513
query35 796 844 749 749
query36 957 1032 896 896
query37 130 103 76 76
query38 4219 4281 4295 4281
query39 1524 1441 1406 1406
query40 220 116 108 108
query41 52 52 54 52
query42 119 112 110 110
query43 536 510 490 490
query44 1294 800 797 797
query45 174 178 164 164
query46 884 1060 652 652
query47 1858 1869 1796 1796
query48 408 425 317 317
query49 702 494 444 444
query50 727 772 430 430
query51 4228 4311 4190 4190
query52 110 106 100 100
query53 233 266 187 187
query54 476 512 414 414
query55 84 89 87 87
query56 270 277 287 277
query57 1154 1226 1115 1115
query58 239 239 261 239
query59 2763 2895 2759 2759
query60 290 300 293 293
query61 162 140 134 134
query62 760 753 682 682
query63 237 198 194 194
query64 1604 1126 696 696
query65 3320 3185 3251 3185
query66 738 404 290 290
query67 16106 15631 15532 15532
query68 5428 791 512 512
query69 524 308 263 263
query70 1149 1155 1132 1132
query71 462 295 276 276
query72 5969 3641 3711 3641
query73 1097 753 346 346
query74 8967 9199 8834 8834
query75 3222 3166 2715 2715
query76 3941 1169 740 740
query77 559 364 340 340
query78 9980 10208 9363 9363
query79 2589 818 600 600
query80 650 523 448 448
query81 497 279 234 234
query82 288 154 121 121
query83 168 178 163 163
query84 301 109 73 73
query85 835 340 312 312
query86 393 313 280 280
query87 4455 4519 4305 4305
query88 3618 2227 2204 2204
query89 413 320 279 279
query90 1782 193 193 193
query91 134 139 107 107
query92 71 61 54 54
query93 2235 1004 581 581
query94 657 410 294 294
query95 356 271 259 259
query96 485 568 268 268
query97 2782 2835 2737 2737
query98 246 208 207 207
query99 1331 1432 1319 1319
Total cold run time: 295021 ms
Total hot run time: 190482 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-3055728033
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-3061346371
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-3051932128
# BE UT Coverage Report
Increment line coverage `3.75% (26/693)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/da950276a9235965a5acc2d5ba77e2579c87ae70_da950276a9235965a5acc2d5ba77e2579c87ae70/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/da950276a9235965a5acc2d5ba77e2579c87ae70_da950276a9235965a5acc2d5ba77e2579c87ae70/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 56.97% (15501/27208) |
| Line Coverage | 45.80% (138185/301707) |
| Region Coverage | 35.29% (103430/293048) |
| Branch Coverage | 37.71% (45475/120605) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-3044425589
<details>
<summary>ClickBench: <b>Total hot run time: 29.37 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit bb350103fa66a6cc5409e2a4761ac6cb9a96d5fd, data reload: false
query1 0.04 0.04 0.04
query2 0.08 0.05 0.04
query3 0.25 0.07 0.07
query4 1.62 0.10 0.11
query5 0.45 0.44 0.41
query6 1.17 0.65 0.66
query7 0.03 0.02 0.02
query8 0.04 0.03 0.04
query9 0.61 0.51 0.52
query10 0.58 0.57 0.57
query11 0.16 0.11 0.11
query12 0.15 0.11 0.12
query13 0.62 0.60 0.61
query14 0.79 0.82 0.81
query15 0.89 0.88 0.86
query16 0.39 0.38 0.40
query17 1.07 1.06 1.03
query18 0.23 0.22 0.21
query19 1.92 1.80 1.80
query20 0.02 0.01 0.01
query21 15.40 0.87 0.54
query22 0.77 1.14 0.68
query23 14.98 1.40 0.67
query24 6.76 1.02 0.64
query25 0.50 0.16 0.06
query26 0.59 0.17 0.13
query27 0.08 0.06 0.04
query28 9.26 0.90 0.45
query29 12.58 4.00 3.28
query30 0.26 0.10 0.06
query31 2.82 0.60 0.39
query32 3.23 0.58 0.47
query33 3.08 3.14 3.15
query34 16.19 5.40 4.79
query35 4.82 4.83 4.87
query36 0.70 0.52 0.50
query37 0.09 0.07 0.07
query38 0.06 0.04 0.04
query39 0.04 0.03 0.02
query40 0.17 0.14 0.13
query41 0.08 0.02 0.03
query42 0.04 0.03 0.03
query43 0.04 0.03 0.03
Total cold run time: 103.65 s
Total hot run time: 29.37 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-3044550980
# Cloud UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/269e246b2133708ac5567562989edaa162f27a63_269e246b2133708ac5567562989edaa162f27a63_cloud/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/269e246b2133708ac5567562989edaa162f27a63_269e246b2133708ac5567562989edaa162f27a63_cloud/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 83.02% (1222/1472) |
| Line Coverage | 67.49% (21162/31355) |
| Region Coverage | 67.25% (10540/15673) |
| Branch Coverage | 56.57% (5543/9798) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix incorrect result when there are insert after delete for same key in one load [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2408475555
<details>
<summary>TPC-H: <b>Total hot run time: 40999 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 79fe26c74dbc3442b2b7b76045ec77054fcb9a10, data reload: false
------ Round 1 ----------------------------------
q1 17607 7641 7313 7313
q2 2020 296 297 296
q3 12122 1077 1199 1077
q4 10576 807 829 807
q5 7771 2904 2832 2832
q6 235 150 149 149
q7 1028 649 614 614
q8 9349 1971 1961 1961
q9 6530 6450 6356 6356
q10 7010 2337 2311 2311
q11 438 248 263 248
q12 427 222 214 214
q13 17783 2989 3005 2989
q14 236 223 214 214
q15 581 526 523 523
q16 654 579 591 579
q17 986 508 536 508
q18 7282 6628 6697 6628
q19 1351 1013 1015 1013
q20 483 204 195 195
q21 3925 3196 3232 3196
q22 1089 1019 976 976
Total cold run time: 109483 ms
Total hot run time: 40999 ms
----- Round 2, with runtime_filter_mode=off -----
q1 7298 7248 7274 7248
q2 325 235 235 235
q3 3022 2971 2954 2954
q4 2105 1846 1823 1823
q5 5795 5762 5737 5737
q6 229 141 149 141
q7 2299 1838 1847 1838
q8 3345 3584 3425 3425
q9 8951 8890 8869 8869
q10 3572 3543 3556 3543
q11 587 514 497 497
q12 826 621 652 621
q13 11172 3210 3135 3135
q14 312 286 283 283
q15 579 546 532 532
q16 705 653 632 632
q17 1861 1592 1607 1592
q18 8275 7796 7444 7444
q19 1703 1412 1541 1412
q20 2098 1877 1896 1877
q21 5577 5413 5509 5413
q22 1146 1061 1057 1057
Total cold run time: 71782 ms
Total hot run time: 60308 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix incorrect result when there are insert after delete for same key in one load [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2408546726
<details>
<summary>TPC-H: <b>Total hot run time: 40756 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 02cb8af6bdf15fd2a892f3a48fa8c8b4446761a7, data reload: false
------ Round 1 ----------------------------------
q1 17571 7956 7273 7273
q2 2024 288 271 271
q3 12151 1041 1145 1041
q4 10583 773 706 706
q5 7767 2876 2827 2827
q6 246 156 147 147
q7 1002 633 624 624
q8 9341 1933 2020 1933
q9 6562 6454 6439 6439
q10 6975 2329 2321 2321
q11 449 247 245 245
q12 414 215 216 215
q13 17796 2981 3010 2981
q14 241 209 218 209
q15 568 534 525 525
q16 665 585 588 585
q17 985 571 555 555
q18 7248 6579 6839 6579
q19 1351 964 1057 964
q20 484 201 199 199
q21 3980 3108 3294 3108
q22 1083 1009 1026 1009
Total cold run time: 109486 ms
Total hot run time: 40756 ms
----- Round 2, with runtime_filter_mode=off -----
q1 7263 7218 7239 7218
q2 330 236 233 233
q3 3063 2920 2952 2920
q4 2084 1878 1825 1825
q5 5785 5785 5763 5763
q6 233 141 144 141
q7 2257 1868 1794 1794
q8 3398 3570 3412 3412
q9 8954 8889 8869 8869
q10 3616 3524 3482 3482
q11 580 493 513 493
q12 815 636 658 636
q13 11153 3195 3185 3185
q14 301 284 277 277
q15 595 528 525 525
q16 694 656 649 649
q17 1857 1648 1640 1640
q18 8189 7842 7503 7503
q19 1712 1512 1449 1449
q20 2108 1879 1928 1879
q21 5411 5409 5414 5409
q22 1127 1044 1049 1044
Total cold run time: 71525 ms
Total hot run time: 60346 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix incorrect result when there are insert after delete for same key in one load [doris]
Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2408534938
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix incorrect result when there are insert after delete for same key in one load [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2408478990
<details>
<summary>TPC-DS: <b>Total hot run time: 192400 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 79fe26c74dbc3442b2b7b76045ec77054fcb9a10, data reload: false
query1 947 404 412 404
query2 6268 2185 2044 2044
query3 8684 191 200 191
query4 33760 23696 23760 23696
query5 3616 500 468 468
query6 272 168 179 168
query7 4212 305 315 305
query8 290 226 226 226
query9 9293 2679 2676 2676
query10 463 278 266 266
query11 17811 15387 15216 15216
query12 156 106 96 96
query13 1583 472 437 437
query14 9590 7462 7457 7457
query15 246 166 178 166
query16 8042 454 489 454
query17 1634 604 559 559
query18 2088 318 321 318
query19 369 146 145 145
query20 121 107 112 107
query21 213 110 103 103
query22 4946 4362 4529 4362
query23 34854 34245 34311 34245
query24 10977 2867 2848 2848
query25 651 406 400 400
query26 1511 160 158 158
query27 2589 307 292 292
query28 7684 2410 2409 2409
query29 954 426 428 426
query30 262 158 155 155
query31 1023 795 793 793
query32 100 53 55 53
query33 768 303 314 303
query34 908 516 501 501
query35 879 738 736 736
query36 1109 940 932 932
query37 148 84 85 84
query38 4035 3939 3881 3881
query39 1545 1449 1402 1402
query40 255 99 98 98
query41 48 45 44 44
query42 120 95 96 95
query43 556 496 493 493
query44 1217 806 840 806
query45 200 166 159 159
query46 1143 731 713 713
query47 1933 1843 1850 1843
query48 417 358 341 341
query49 981 421 417 417
query50 812 415 414 414
query51 7139 6944 6842 6842
query52 99 86 88 86
query53 253 184 181 181
query54 1401 470 481 470
query55 77 80 79 79
query56 307 272 261 261
query57 1236 1187 1191 1187
query58 256 251 303 251
query59 3281 3221 3028 3028
query60 317 304 287 287
query61 125 119 118 118
query62 854 665 666 665
query63 225 197 190 190
query64 5112 669 601 601
query65 3306 3168 3203 3168
query66 1381 295 326 295
query67 16016 15729 15562 15562
query68 4419 564 562 562
query69 508 295 302 295
query70 1135 1066 1125 1066
query71 369 264 272 264
query72 7516 3927 3928 3927
query73 761 334 344 334
query74 10190 9073 9087 9073
query75 4011 2674 2715 2674
query76 3237 832 878 832
query77 647 330 315 315
query78 10462 9627 9513 9513
query79 1486 590 592 590
query80 2654 432 452 432
query81 596 245 247 245
query82 732 137 138 137
query83 321 142 141 141
query84 277 84 75 75
query85 1698 299 285 285
query86 449 293 299 293
query87 4520 4378 4278 4278
query88 3215 2396 2368 2368
query89 410 288 284 284
query90 2149 185 180 180
query91 138 105 106 105
query92 64 47 48 47
query93 1172 526 538 526
query94 1238 286 295 286
query95 354 255 251 251
query96 608 278 281 278
query97 3248 3130 3122 3122
query98 219 205 190 190
query99 1534 1256 1307 1256
Total cold run time: 304005 ms
Total hot run time: 192400 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix incorrect result when there are insert after delete for same key in one load [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring commented on code in PR #41701:
URL: https://github.com/apache/doris/pull/41701#discussion_r1802405826
##########
be/src/olap/memtable.cpp:
##########
@@ -578,6 +546,200 @@ void MemTable::_aggregate() {
}
}
+template <bool is_final>
+void MemTable::_aggregate_for_flexible_partial_update_without_seq_col(
+ const vectorized::ColumnsWithTypeAndName& block_data,
+ vectorized::MutableBlock& mutable_block, std::vector<RowInBlock*>& temp_row_in_blocks) {
+ RowInBlock* prev_row = nullptr;
+ int row_pos = -1;
+ auto& skip_bitmaps = assert_cast<vectorized::ColumnBitmap*>(
+ mutable_block.mutable_columns()[_skip_bitmap_col_idx].get())
+ ->get_data();
+ auto& delete_signs = assert_cast<vectorized::ColumnInt8*>(
+ mutable_block.mutable_columns()[_delete_sign_col_idx].get())
+ ->get_data();
+ RowInBlock* row_with_delete_sign {nullptr};
+ RowInBlock* row_without_delete_sign {nullptr};
+
+ auto finalize_rows = [&]() {
+ if (row_with_delete_sign != nullptr) {
+ temp_row_in_blocks.push_back(row_with_delete_sign);
+ _finalize_one_row<is_final, false>(row_with_delete_sign, block_data, ++row_pos);
+ row_with_delete_sign = nullptr;
+ }
+ if (row_without_delete_sign != nullptr) {
+ temp_row_in_blocks.push_back(row_without_delete_sign);
+ _finalize_one_row<is_final, false>(row_without_delete_sign, block_data, ++row_pos);
+ row_without_delete_sign = nullptr;
+ }
+ _arena->clear();
Review Comment:
It seems that, row_with_delete_sign and row_without_delete sign can not be set together. We can use a pair to replace two vars.
std::pair<RowInBlock *, bool>, bool indicates it is has delete sign.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix incorrect result when there are insert after delete for same key in one load [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2422375614
<details>
<summary>ClickBench: <b>Total hot run time: 33.13 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e1f3877e946fc22d7a55d804f2346266653ae2fd, data reload: false
query1 0.04 0.04 0.04
query2 0.06 0.03 0.03
query3 0.22 0.06 0.06
query4 1.65 0.10 0.11
query5 0.51 0.51 0.52
query6 1.13 0.72 0.73
query7 0.02 0.01 0.01
query8 0.04 0.03 0.02
query9 0.56 0.50 0.50
query10 0.55 0.56 0.55
query11 0.14 0.11 0.11
query12 0.14 0.10 0.12
query13 0.61 0.60 0.59
query14 2.72 2.73 2.80
query15 0.90 0.83 0.82
query16 0.38 0.37 0.38
query17 1.06 1.02 1.04
query18 0.23 0.22 0.21
query19 1.96 1.77 2.03
query20 0.01 0.01 0.01
query21 15.37 0.61 0.59
query22 2.56 2.27 2.44
query23 17.04 0.95 1.03
query24 2.86 1.00 0.87
query25 0.25 0.12 0.17
query26 0.35 0.13 0.14
query27 0.04 0.04 0.04
query28 11.01 1.09 1.06
query29 12.52 3.19 3.16
query30 0.24 0.07 0.06
query31 2.88 0.38 0.39
query32 3.25 0.46 0.46
query33 2.95 3.02 3.04
query34 16.75 4.47 4.49
query35 4.49 4.51 4.49
query36 0.68 0.51 0.49
query37 0.09 0.06 0.06
query38 0.05 0.03 0.03
query39 0.03 0.02 0.02
query40 0.17 0.12 0.13
query41 0.08 0.03 0.02
query42 0.03 0.02 0.02
query43 0.04 0.03 0.03
Total cold run time: 106.66 s
Total hot run time: 33.13 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2556322504
TeamCity be ut coverage result:
Function Coverage: 38.84% (10131/26081)
Line Coverage: 29.77% (85243/286358)
Region Coverage: 28.86% (43471/150613)
Branch Coverage: 25.40% (22170/87296)
Coverage Report: http://coverage.selectdb-in.cc/coverage/c9f2159f3b4ce84a1b67858cefd8b3ceced09e37_c9f2159f3b4ce84a1b67858cefd8b3ceced09e37/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2561911445
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2493494123
TeamCity be ut coverage result:
Function Coverage: 38.28% (9978/26068)
Line Coverage: 29.38% (83505/284194)
Region Coverage: 28.53% (42966/150580)
Branch Coverage: 25.13% (21833/86892)
Coverage Report: http://coverage.selectdb-in.cc/coverage/61a7fd52e74446b3f27eccd533514d31d5b431bf_61a7fd52e74446b3f27eccd533514d31d5b431bf/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #41701:
URL: https://github.com/apache/doris/pull/41701#discussion_r1833931935
##########
be/src/olap/partial_update_info.cpp:
##########
@@ -658,4 +684,431 @@ Status FlexibleReadPlan::fill_non_primary_key_columns_for_row_store(
return Status::OK();
}
+BlockAggregator::BlockAggregator(segment_v2::VerticalSegmentWriter& vertical_segment_writer)
+ : _writer(vertical_segment_writer), _tablet_schema(*_writer._tablet_schema) {}
+
+void BlockAggregator::merge_one_row(vectorized::MutableBlock& dst_block,
+ vectorized::Block* src_block, int64_t rid,
+ BitmapValue& skip_bitmap) {
+ for (size_t cid {_tablet_schema.num_key_columns()}; cid < _tablet_schema.num_columns(); cid++) {
+ if (cid == _tablet_schema.skip_bitmap_col_idx()) {
+ auto& cur_skip_bitmap =
+ assert_cast<vectorized::ColumnBitmap*>(dst_block.mutable_columns()[cid].get())
+ ->get_data()
+ .back();
+ const auto& new_row_skip_bitmap =
+ assert_cast<vectorized::ColumnBitmap*>(
+ src_block->get_by_position(cid).column->assume_mutable().get())
+ ->get_data()[rid];
+ cur_skip_bitmap &= new_row_skip_bitmap;
+ continue;
+ }
+ if (!skip_bitmap.contains(_tablet_schema.column(cid).unique_id())) {
+ dst_block.mutable_columns()[cid]->pop_back(1);
+ dst_block.mutable_columns()[cid]->insert_from(*src_block->get_by_position(cid).column,
+ rid);
+ }
+ }
+ VLOG_DEBUG << fmt::format("merge a row, after merge, output_block.rows()={}, state: {}",
+ dst_block.rows(), _state.to_string());
+}
+
+void BlockAggregator::append_one_row(vectorized::MutableBlock& dst_block,
+ vectorized::Block* src_block, int64_t rid) {
+ dst_block.add_row(src_block, rid);
+ _state.rows++;
+ VLOG_DEBUG << fmt::format("append a new row, after append, output_block.rows()={}, state: {}",
+ dst_block.rows(), _state.to_string());
+}
+
+void BlockAggregator::remove_last_n_rows(vectorized::MutableBlock& dst_block, int n) {
+ if (n > 0) {
+ for (size_t cid {0}; cid < _tablet_schema.num_columns(); cid++) {
+ DCHECK_GE(dst_block.mutable_columns()[cid]->size(), n);
+ dst_block.mutable_columns()[cid]->pop_back(n);
+ }
+ }
+}
+
+void BlockAggregator::append_or_merge_row(vectorized::MutableBlock& dst_block,
+ vectorized::Block* src_block, int64_t rid,
+ BitmapValue& skip_bitmap, bool have_delete_sign) {
+ if (have_delete_sign) {
+ // remove all the previous batched rows
+ remove_last_n_rows(dst_block, _state.rows);
+ _state.rows = 0;
+ _state.has_row_with_delete_sign = true;
+
+ append_one_row(dst_block, src_block, rid);
+ } else {
+ if (_state.should_merge()) {
+ merge_one_row(dst_block, src_block, rid, skip_bitmap);
+ } else {
+ append_one_row(dst_block, src_block, rid);
+ }
+ }
+};
+
+Status BlockAggregator::aggregate_rows(
Review Comment:
warning: function 'aggregate_rows' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
Status BlockAggregator::aggregate_rows(
^
```
<details>
<summary>Additional context</summary>
**be/src/olap/partial_update_info.cpp:751:** 84 lines including whitespace and comments (threshold 80)
```cpp
Status BlockAggregator::aggregate_rows(
^
```
</details>
##########
be/src/olap/partial_update_info.cpp:
##########
@@ -658,4 +684,431 @@
return Status::OK();
}
+BlockAggregator::BlockAggregator(segment_v2::VerticalSegmentWriter& vertical_segment_writer)
+ : _writer(vertical_segment_writer), _tablet_schema(*_writer._tablet_schema) {}
+
+void BlockAggregator::merge_one_row(vectorized::MutableBlock& dst_block,
+ vectorized::Block* src_block, int64_t rid,
+ BitmapValue& skip_bitmap) {
+ for (size_t cid {_tablet_schema.num_key_columns()}; cid < _tablet_schema.num_columns(); cid++) {
+ if (cid == _tablet_schema.skip_bitmap_col_idx()) {
+ auto& cur_skip_bitmap =
+ assert_cast<vectorized::ColumnBitmap*>(dst_block.mutable_columns()[cid].get())
+ ->get_data()
+ .back();
+ const auto& new_row_skip_bitmap =
+ assert_cast<vectorized::ColumnBitmap*>(
+ src_block->get_by_position(cid).column->assume_mutable().get())
+ ->get_data()[rid];
+ cur_skip_bitmap &= new_row_skip_bitmap;
+ continue;
+ }
+ if (!skip_bitmap.contains(_tablet_schema.column(cid).unique_id())) {
+ dst_block.mutable_columns()[cid]->pop_back(1);
+ dst_block.mutable_columns()[cid]->insert_from(*src_block->get_by_position(cid).column,
+ rid);
+ }
+ }
+ VLOG_DEBUG << fmt::format("merge a row, after merge, output_block.rows()={}, state: {}",
+ dst_block.rows(), _state.to_string());
+}
+
+void BlockAggregator::append_one_row(vectorized::MutableBlock& dst_block,
+ vectorized::Block* src_block, int64_t rid) {
+ dst_block.add_row(src_block, rid);
+ _state.rows++;
+ VLOG_DEBUG << fmt::format("append a new row, after append, output_block.rows()={}, state: {}",
+ dst_block.rows(), _state.to_string());
+}
+
+void BlockAggregator::remove_last_n_rows(vectorized::MutableBlock& dst_block, int n) {
+ if (n > 0) {
+ for (size_t cid {0}; cid < _tablet_schema.num_columns(); cid++) {
+ DCHECK_GE(dst_block.mutable_columns()[cid]->size(), n);
+ dst_block.mutable_columns()[cid]->pop_back(n);
+ }
+ }
+}
+
+void BlockAggregator::append_or_merge_row(vectorized::MutableBlock& dst_block,
+ vectorized::Block* src_block, int64_t rid,
+ BitmapValue& skip_bitmap, bool have_delete_sign) {
+ if (have_delete_sign) {
+ // remove all the previous batched rows
+ remove_last_n_rows(dst_block, _state.rows);
+ _state.rows = 0;
+ _state.has_row_with_delete_sign = true;
+
+ append_one_row(dst_block, src_block, rid);
+ } else {
+ if (_state.should_merge()) {
+ merge_one_row(dst_block, src_block, rid, skip_bitmap);
+ } else {
+ append_one_row(dst_block, src_block, rid);
+ }
+ }
+};
+
+Status BlockAggregator::aggregate_rows(
+ vectorized::MutableBlock& output_block, vectorized::Block* block, int64_t start,
+ int64_t end, std::string key, std::vector<BitmapValue>* skip_bitmaps,
+ const signed char* delete_signs, vectorized::IOlapColumnDataAccessor* seq_column,
+ const std::vector<RowsetSharedPtr>& specified_rowsets,
+ std::vector<std::unique_ptr<SegmentCacheHandle>>& segment_caches) {
+ VLOG_DEBUG << fmt::format("merge rows in range=[{}-{})", start, end);
+ if (end - start == 1) {
+ output_block.add_row(block, start);
+ VLOG_DEBUG << fmt::format("append a row directly, rid={}", start);
+ return Status::OK();
+ }
+
+ auto seq_col_unique_id = _tablet_schema.column(_tablet_schema.sequence_col_idx()).unique_id();
+ auto delete_sign_col_unique_id =
+ _tablet_schema.column(_tablet_schema.delete_sign_idx()).unique_id();
+
+ _state.reset();
+
+ RowLocation loc;
+ RowsetSharedPtr rowset;
+ std::string previous_encoded_seq_value {};
+ Status st = _writer._tablet->lookup_row_key(
+ key, &_tablet_schema, false, specified_rowsets, &loc, _writer._mow_context->max_version,
+ segment_caches, &rowset, true, &previous_encoded_seq_value);
+ int64_t pos = start;
+ bool is_expected_st = (st.is<ErrorCode::KEY_NOT_FOUND>() || st.ok());
+ DCHECK(is_expected_st || st.is<ErrorCode::MEM_LIMIT_EXCEEDED>())
+ << "[BlockAggregator::aggregate_rows] unexpected error status while lookup_row_key:"
+ << st;
+ if (!is_expected_st) {
+ return st;
+ }
+
+ std::string cur_seq_val;
+ if (st.ok()) {
+ for (pos = start; pos < end; pos++) {
+ auto& skip_bitmap = skip_bitmaps->at(pos);
+ bool row_has_sequence_col = (!skip_bitmap.contains(seq_col_unique_id));
+ // Discard all the rows whose seq value is smaller than previous_encoded_seq_value.
+ if (row_has_sequence_col) {
+ std::string seq_val {};
+ _writer._encode_seq_column(seq_column, pos, &seq_val);
+ if (Slice {seq_val}.compare(Slice {previous_encoded_seq_value}) < 0) {
+ continue;
+ }
+ cur_seq_val = std::move(seq_val);
+ break;
+ }
+ cur_seq_val = std::move(previous_encoded_seq_value);
+ break;
+ }
+ } else {
+ pos = start;
+ auto& skip_bitmap = skip_bitmaps->at(pos);
+ bool row_has_sequence_col = (!skip_bitmap.contains(seq_col_unique_id));
+ if (row_has_sequence_col) {
+ std::string seq_val {};
+ // for rows that don't specify seqeunce col, seq_val will be encoded to minial value
+ _writer._encode_seq_column(seq_column, pos, &seq_val);
+ cur_seq_val = std::move(seq_val);
+ } else {
+ cur_seq_val.clear();
+ RETURN_IF_ERROR(_writer._generate_encoded_default_seq_value(
+ _tablet_schema, *_writer._opts.rowset_ctx->partial_update_info, &cur_seq_val));
+ }
+ }
+ VLOG_DEBUG << fmt::format("start pos={}, output_block.rows()={}", pos, output_block.rows());
+
+ for (int64_t rid {pos}; rid < end; rid++) {
+ auto& skip_bitmap = skip_bitmaps->at(rid);
+ bool row_has_sequence_col = (!skip_bitmap.contains(seq_col_unique_id));
+ bool have_delete_sign =
+ (!skip_bitmap.contains(delete_sign_col_unique_id) && delete_signs[rid] != 0);
+ if (!row_has_sequence_col) {
+ append_or_merge_row(output_block, block, rid, skip_bitmap, have_delete_sign);
+ } else {
+ std::string seq_val {};
+ _writer._encode_seq_column(seq_column, rid, &seq_val);
+ if (Slice {seq_val}.compare(Slice {cur_seq_val}) >= 0) {
+ append_or_merge_row(output_block, block, rid, skip_bitmap, have_delete_sign);
+ cur_seq_val = std::move(seq_val);
+ } else {
+ VLOG_DEBUG << fmt::format(
+ "skip rid={} becasue its seq value is lower than the previous", rid);
+ }
+ }
+ }
+ return Status::OK();
+};
+
+Status BlockAggregator::aggregate_for_sequence_column(
+ vectorized::Block* block, size_t num_rows,
+ const std::vector<vectorized::IOlapColumnDataAccessor*>& key_columns,
+ vectorized::IOlapColumnDataAccessor* seq_column,
+ const std::vector<RowsetSharedPtr>& specified_rowsets,
+ std::vector<std::unique_ptr<SegmentCacheHandle>>& segment_caches) {
+ DCHECK_EQ(block->columns(), _tablet_schema.num_columns());
+ VLOG_DEBUG << fmt::format("BlockAggregator::aggregate_for_sequence_column enter: block:{}\n",
+ block->dump_data());
+ // the process logic here is the same as MemTable::_aggregate_for_flexible_partial_update_without_seq_col()
+ // after this function, there will be at most 2 rows for a specified key
+ std::vector<BitmapValue>* skip_bitmaps =
+ &(assert_cast<vectorized::ColumnBitmap*>(
+ block->get_by_position(_tablet_schema.skip_bitmap_col_idx())
+ .column->assume_mutable()
+ .get())
+ ->get_data());
+ const auto* delete_signs = BaseTablet::get_delete_sign_column_data(*block, num_rows);
+
+ auto filtered_block = _tablet_schema.create_block();
+ vectorized::MutableBlock output_block =
+ vectorized::MutableBlock::build_mutable_block(&filtered_block);
+
+ int same_key_rows {0};
+ std::string previous_key {};
+ for (size_t block_pos {0}; block_pos < num_rows; block_pos++) {
+ std::string key = _writer._full_encode_keys(key_columns, block_pos);
+ if (block_pos > 0 && previous_key == key) {
+ same_key_rows++;
+ } else {
+ if (same_key_rows > 0) {
+ RETURN_IF_ERROR(aggregate_rows(output_block, block, block_pos - same_key_rows,
+ block_pos, std::move(previous_key), skip_bitmaps,
+ delete_signs, seq_column, specified_rowsets,
+ segment_caches));
+ }
+ same_key_rows = 1;
+ }
+ previous_key = std::move(key);
+ }
+ if (same_key_rows > 0) {
+ RETURN_IF_ERROR(aggregate_rows(output_block, block, num_rows - same_key_rows, num_rows,
+ std::move(previous_key), skip_bitmaps, delete_signs,
+ seq_column, specified_rowsets, segment_caches));
+ }
+
+ block->swap(output_block.to_block());
+ VLOG_DEBUG << fmt::format(
+ "BlockAggregator::aggregate_for_sequence_column after: data.block:{}\n",
+ block->dump_data());
+ return Status::OK();
+}
+
+Status BlockAggregator::fill_sequence_column(vectorized::Block* block, size_t num_rows,
+ const FixedReadPlan& read_plan,
+ std::vector<BitmapValue>& skip_bitmaps) {
+ DCHECK(_tablet_schema.has_sequence_col());
+ std::vector<uint32_t> cids {static_cast<uint32_t>(_tablet_schema.sequence_col_idx())};
+ auto seq_col_unique_id = _tablet_schema.column(_tablet_schema.sequence_col_idx()).unique_id();
+
+ auto seq_col_block = _tablet_schema.create_block_by_cids(cids);
+ auto tmp_block = _tablet_schema.create_block_by_cids(cids);
+ std::map<uint32_t, uint32_t> read_index;
+ RETURN_IF_ERROR(read_plan.read_columns_by_plan(_tablet_schema, cids, _writer._rsid_to_rowset,
+ seq_col_block, &read_index));
+
+ auto new_seq_col_ptr = tmp_block.get_by_position(0).column->assume_mutable();
+ const auto& old_seq_col_ptr = *seq_col_block.get_by_position(0).column;
+ const auto& cur_seq_col_ptr = *block->get_by_position(_tablet_schema.sequence_col_idx()).column;
+ for (size_t block_pos {0}; block_pos < num_rows; block_pos++) {
+ if (read_index.contains(block_pos)) {
+ new_seq_col_ptr->insert_from(old_seq_col_ptr, read_index[block_pos]);
+ skip_bitmaps[block_pos].remove(seq_col_unique_id);
+ } else {
+ new_seq_col_ptr->insert_from(cur_seq_col_ptr, block_pos);
+ }
+ }
+ block->replace_by_position(_tablet_schema.sequence_col_idx(), std::move(new_seq_col_ptr));
+ return Status::OK();
+}
+
+Status BlockAggregator::aggregate_for_insert_after_delete(
Review Comment:
warning: function 'aggregate_for_insert_after_delete' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
Status BlockAggregator::aggregate_for_insert_after_delete(
^
```
<details>
<summary>Additional context</summary>
**be/src/olap/partial_update_info.cpp:923:** 96 lines including whitespace and comments (threshold 80)
```cpp
Status BlockAggregator::aggregate_for_insert_after_delete(
^
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2594694299
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2664859823
<details>
<summary>ClickBench: <b>Total hot run time: 30.24 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f94d45e93c29681112808b185d09bb4ea8965344, data reload: false
query1 0.03 0.03 0.04
query2 0.08 0.03 0.03
query3 0.24 0.07 0.06
query4 1.65 0.10 0.10
query5 0.41 0.42 0.42
query6 1.16 0.66 0.65
query7 0.02 0.02 0.02
query8 0.04 0.03 0.03
query9 0.58 0.51 0.55
query10 0.56 0.58 0.57
query11 0.16 0.11 0.10
query12 0.14 0.10 0.11
query13 0.62 0.62 0.59
query14 2.70 2.72 2.71
query15 0.92 0.86 0.85
query16 0.38 0.39 0.36
query17 1.01 1.03 1.04
query18 0.22 0.19 0.19
query19 1.88 1.98 1.86
query20 0.01 0.02 0.01
query21 15.36 0.92 0.54
query22 0.75 1.17 0.77
query23 14.84 1.38 0.61
query24 11.56 1.24 0.52
query25 0.47 0.34 0.07
query26 0.96 0.18 0.15
query27 0.04 0.05 0.04
query28 5.98 0.78 0.43
query29 12.55 3.96 3.28
query30 0.25 0.08 0.06
query31 2.81 0.59 0.38
query32 3.23 0.55 0.46
query33 2.98 2.95 3.02
query34 15.82 5.16 4.48
query35 4.51 4.50 4.52
query36 0.68 0.49 0.47
query37 0.09 0.06 0.06
query38 0.04 0.04 0.04
query39 0.03 0.03 0.02
query40 0.18 0.13 0.14
query41 0.09 0.02 0.02
query42 0.03 0.02 0.02
query43 0.04 0.03 0.02
Total cold run time: 106.1 s
Total hot run time: 30.24 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2642249944
<details>
<summary>ClickBench: <b>Total hot run time: 31.21 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ebcaf82a4f4c0679d1c2fcdb77d2a5355d5a12b6, data reload: false
query1 0.04 0.03 0.03
query2 0.10 0.04 0.04
query3 0.27 0.06 0.05
query4 1.60 0.07 0.07
query5 0.40 0.40 0.39
query6 1.17 0.65 0.65
query7 0.02 0.02 0.02
query8 0.05 0.05 0.05
query9 0.61 0.52 0.52
query10 0.58 0.60 0.57
query11 0.25 0.12 0.13
query12 0.25 0.14 0.14
query13 0.63 0.62 0.60
query14 2.68 2.73 2.70
query15 0.99 0.87 0.88
query16 0.38 0.36 0.37
query17 1.02 1.00 1.02
query18 0.17 0.19 0.18
query19 1.95 1.83 2.01
query20 0.01 0.02 0.01
query21 15.36 0.95 0.66
query22 0.92 0.99 0.85
query23 14.69 1.46 0.73
query24 5.46 0.56 0.28
query25 0.16 0.09 0.09
query26 0.55 0.21 0.18
query27 0.08 0.08 0.09
query28 10.97 1.20 0.54
query29 12.89 4.22 3.45
query30 0.27 0.08 0.06
query31 2.82 0.60 0.42
query32 3.24 0.58 0.50
query33 3.02 3.01 3.07
query34 16.52 5.21 4.48
query35 4.56 4.50 4.52
query36 0.62 0.51 0.51
query37 0.21 0.18 0.17
query38 0.17 0.16 0.15
query39 0.05 0.04 0.04
query40 0.19 0.16 0.16
query41 0.11 0.05 0.05
query42 0.07 0.05 0.05
query43 0.05 0.05 0.05
Total cold run time: 106.15 s
Total hot run time: 31.21 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-3044033348
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-3051550836
# FE UT Coverage Report
Increment line coverage ` 100.00% (1/1)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/41701_da950276a9235965a5acc2d5ba77e2579c87ae70/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/41701_da950276a9235965a5acc2d5ba77e2579c87ae70/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-3055871873
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-3057398939
# BE UT Coverage Report
Increment line coverage `3.85% (26/675)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/49c2fe8f4ea4a7a466b4a04119035ae5a1a9ceaf_49c2fe8f4ea4a7a466b4a04119035ae5a1a9ceaf/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/49c2fe8f4ea4a7a466b4a04119035ae5a1a9ceaf_49c2fe8f4ea4a7a466b4a04119035ae5a1a9ceaf/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 57.02% (15508/27198) |
| Line Coverage | 45.84% (138416/301981) |
| Region Coverage | 35.28% (103581/293576) |
| Branch Coverage | 37.71% (45576/120872) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-3060337564
# Cloud UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/8139e2d8e542a5e69903ac8e087381bed76cc5fe_8139e2d8e542a5e69903ac8e087381bed76cc5fe_cloud/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/8139e2d8e542a5e69903ac8e087381bed76cc5fe_8139e2d8e542a5e69903ac8e087381bed76cc5fe_cloud/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 81.91% (1250/1526) |
| Line Coverage | 67.67% (21349/31550) |
| Region Coverage | 67.30% (10597/15746) |
| Branch Coverage | 56.78% (5576/9820) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-3056900431
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-3057114375
<details>
<summary>TPC-H: <b>Total hot run time: 33282 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 49c2fe8f4ea4a7a466b4a04119035ae5a1a9ceaf, data reload: false
------ Round 1 ----------------------------------
q1 17583 5143 5054 5054
q2 1926 277 199 199
q3 10316 1294 722 722
q4 10247 1020 506 506
q5 8073 2403 2347 2347
q6 176 158 129 129
q7 933 746 595 595
q8 9312 1335 1068 1068
q9 6896 5118 5134 5118
q10 6873 2377 1949 1949
q11 539 285 267 267
q12 341 362 215 215
q13 17791 3660 3128 3128
q14 233 217 217 217
q15 536 475 482 475
q16 425 428 380 380
q17 583 881 354 354
q18 7623 7168 7166 7166
q19 1307 949 546 546
q20 338 351 213 213
q21 3699 3169 2338 2338
q22 354 312 296 296
Total cold run time: 106104 ms
Total hot run time: 33282 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5070 5261 5075 5075
q2 257 326 214 214
q3 2219 2732 2286 2286
q4 1382 1797 1364 1364
q5 4232 4351 4585 4351
q6 215 164 124 124
q7 2071 2011 1797 1797
q8 2639 2659 2669 2659
q9 7334 7157 7240 7157
q10 3191 3315 2946 2946
q11 592 513 503 503
q12 741 825 630 630
q13 3656 4014 3371 3371
q14 284 300 281 281
q15 520 510 464 464
q16 448 486 445 445
q17 1228 1602 1387 1387
q18 8107 7757 7622 7622
q19 801 888 1065 888
q20 2103 2125 1926 1926
q21 4995 4579 4552 4552
q22 677 590 537 537
Total cold run time: 52762 ms
Total hot run time: 50579 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-3079199136
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix incorrect result when there are insert after delete for same key in one load [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2434613244
<details>
<summary>ClickBench: <b>Total hot run time: 31.57 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7a452bc70491546bb7a6e70313f42b0b80430833, data reload: false
query1 0.04 0.03 0.03
query2 0.07 0.03 0.03
query3 0.24 0.07 0.06
query4 1.64 0.10 0.10
query5 0.41 0.40 0.38
query6 1.15 0.64 0.65
query7 0.02 0.03 0.01
query8 0.04 0.04 0.04
query9 0.55 0.50 0.50
query10 0.56 0.55 0.55
query11 0.14 0.11 0.11
query12 0.13 0.11 0.11
query13 0.62 0.59 0.59
query14 2.81 2.70 2.71
query15 0.92 0.83 0.84
query16 0.39 0.39 0.37
query17 1.06 0.99 0.97
query18 0.21 0.20 0.20
query19 1.96 1.88 1.92
query20 0.02 0.00 0.01
query21 15.36 0.60 0.58
query22 3.26 1.93 1.56
query23 17.07 0.82 0.90
query24 3.11 0.31 1.64
query25 0.27 0.16 0.09
query26 0.41 0.14 0.15
query27 0.05 0.04 0.03
query28 10.64 1.08 1.07
query29 12.52 3.30 3.23
query30 0.24 0.06 0.06
query31 2.87 0.39 0.37
query32 3.27 0.46 0.45
query33 2.96 3.04 3.03
query34 16.96 4.46 4.57
query35 4.59 4.51 4.54
query36 0.65 0.47 0.50
query37 0.09 0.06 0.06
query38 0.04 0.03 0.04
query39 0.03 0.02 0.02
query40 0.16 0.13 0.12
query41 0.08 0.02 0.02
query42 0.03 0.02 0.02
query43 0.04 0.03 0.02
Total cold run time: 107.68 s
Total hot run time: 31.57 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix incorrect result when there are insert after delete for same key in one load [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2434569360
TeamCity be ut coverage result:
Function Coverage: 37.46% (9713/25929)
Line Coverage: 28.72% (80545/280480)
Region Coverage: 28.15% (41651/147978)
Branch Coverage: 24.72% (21161/85616)
Coverage Report: http://coverage.selectdb-in.cc/coverage/7a452bc70491546bb7a6e70313f42b0b80430833_7a452bc70491546bb7a6e70313f42b0b80430833/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix incorrect result when there are insert after delete for same key in one load [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2434575879
<details>
<summary>TPC-H: <b>Total hot run time: 41635 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7a452bc70491546bb7a6e70313f42b0b80430833, data reload: false
------ Round 1 ----------------------------------
q1 17584 7553 7413 7413
q2 2023 273 274 273
q3 12406 1093 1214 1093
q4 10564 841 781 781
q5 7763 3171 3142 3142
q6 236 153 151 151
q7 1047 621 604 604
q8 9351 1986 1961 1961
q9 6540 6542 6430 6430
q10 7056 2424 2439 2424
q11 444 260 240 240
q12 432 228 222 222
q13 17797 3013 3005 3005
q14 251 220 212 212
q15 567 513 528 513
q16 654 601 586 586
q17 994 567 524 524
q18 7341 6757 6703 6703
q19 1353 965 994 965
q20 490 194 186 186
q21 4298 3195 3192 3192
q22 1123 1023 1015 1015
Total cold run time: 110314 ms
Total hot run time: 41635 ms
----- Round 2, with runtime_filter_mode=off -----
q1 7316 7331 7317 7317
q2 341 237 247 237
q3 3074 3003 2978 2978
q4 2178 1817 1860 1817
q5 5776 5769 5807 5769
q6 231 146 144 144
q7 2261 1878 1792 1792
q8 3450 3541 3495 3495
q9 8983 8997 8915 8915
q10 3607 3598 3574 3574
q11 598 489 505 489
q12 885 656 633 633
q13 10716 3216 3175 3175
q14 305 275 287 275
q15 590 542 541 541
q16 684 649 655 649
q17 1877 1626 1631 1626
q18 8277 7925 7667 7667
q19 1763 1456 1393 1393
q20 2125 1894 1881 1881
q21 5634 5606 5516 5516
q22 1130 1060 1100 1060
Total cold run time: 71801 ms
Total hot run time: 60943 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-3044405351
<details>
<summary>TPC-DS: <b>Total hot run time: 187513 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit bb350103fa66a6cc5409e2a4761ac6cb9a96d5fd, data reload: false
query1 1021 392 389 389
query2 6566 1665 1642 1642
query3 6746 217 209 209
query4 26703 23394 23377 23377
query5 4334 597 453 453
query6 298 215 196 196
query7 4625 503 288 288
query8 270 209 218 209
query9 8587 2677 2689 2677
query10 468 344 277 277
query11 15285 15017 14800 14800
query12 157 112 110 110
query13 1649 545 401 401
query14 8647 6092 6080 6080
query15 209 191 173 173
query16 7244 443 269 269
query17 1314 756 610 610
query18 1990 426 315 315
query19 197 198 164 164
query20 126 122 115 115
query21 216 126 108 108
query22 4273 4286 4577 4286
query23 35027 34004 33777 33777
query24 8528 2383 2381 2381
query25 544 449 385 385
query26 1277 267 149 149
query27 2774 510 344 344
query28 4263 2146 2119 2119
query29 762 531 426 426
query30 286 218 191 191
query31 917 862 761 761
query32 70 65 60 60
query33 555 364 285 285
query34 823 869 544 544
query35 587 630 559 559
query36 939 976 869 869
query37 114 104 74 74
query38 4139 4182 4165 4165
query39 1482 1400 1394 1394
query40 208 114 104 104
query41 57 52 48 48
query42 120 115 110 110
query43 491 506 457 457
query44 1382 857 848 848
query45 174 169 165 165
query46 899 1052 639 639
query47 1734 1769 1761 1761
query48 384 434 318 318
query49 724 471 395 395
query50 694 725 410 410
query51 4244 4244 4232 4232
query52 106 114 95 95
query53 227 266 182 182
query54 576 563 536 536
query55 86 80 77 77
query56 345 291 288 288
query57 1186 1180 1114 1114
query58 263 257 255 255
query59 2576 2608 2557 2557
query60 338 321 314 314
query61 142 138 119 119
query62 818 719 677 677
query63 230 187 186 186
query64 4434 1203 857 857
query65 4254 4201 4175 4175
query66 1132 429 300 300
query67 16002 15694 15525 15525
query68 9404 913 541 541
query69 510 311 271 271
query70 1187 1062 1119 1062
query71 481 329 305 305
query72 5539 4725 4861 4725
query73 733 634 360 360
query74 8895 9270 9003 9003
query75 4242 3291 2680 2680
query76 3744 1160 753 753
query77 964 402 293 293
query78 11057 11016 10242 10242
query79 2299 828 581 581
query80 632 507 421 421
query81 460 257 220 220
query82 472 133 97 97
query83 288 249 235 235
query84 333 103 82 82
query85 789 362 306 306
query86 328 286 287 286
query87 4417 4377 4261 4261
query88 3479 2284 2320 2284
query89 394 325 277 277
query90 1922 208 200 200
query91 141 141 106 106
query92 71 57 62 57
query93 1390 943 595 595
query94 688 312 192 192
query95 375 284 283 283
query96 502 576 280 280
query97 2750 2717 2674 2674
query98 223 210 203 203
query99 1472 1399 1263 1263
Total cold run time: 278019 ms
Total hot run time: 187513 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-3044688310
<details>
<summary>ClickBench: <b>Total hot run time: 29.42 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 269e246b2133708ac5567562989edaa162f27a63, data reload: false
query1 0.04 0.03 0.04
query2 0.08 0.04 0.05
query3 0.25 0.08 0.07
query4 1.62 0.10 0.11
query5 0.45 0.42 0.41
query6 1.18 0.65 0.67
query7 0.02 0.02 0.02
query8 0.05 0.03 0.04
query9 0.60 0.51 0.52
query10 0.57 0.59 0.57
query11 0.16 0.11 0.11
query12 0.15 0.12 0.12
query13 0.63 0.61 0.61
query14 0.78 0.83 0.83
query15 0.89 0.88 0.89
query16 0.39 0.38 0.39
query17 1.08 1.06 1.08
query18 0.23 0.22 0.21
query19 1.92 1.81 1.86
query20 0.02 0.02 0.01
query21 15.43 0.87 0.54
query22 0.76 1.25 0.77
query23 14.75 1.39 0.62
query24 6.45 1.91 0.52
query25 0.51 0.20 0.19
query26 0.58 0.16 0.13
query27 0.07 0.05 0.05
query28 9.84 0.90 0.46
query29 12.54 3.91 3.29
query30 0.25 0.10 0.07
query31 2.84 0.60 0.40
query32 3.26 0.56 0.49
query33 3.06 3.10 3.16
query34 16.18 5.39 4.77
query35 4.84 4.85 4.79
query36 0.70 0.50 0.47
query37 0.09 0.06 0.07
query38 0.05 0.04 0.03
query39 0.03 0.02 0.02
query40 0.17 0.15 0.13
query41 0.09 0.02 0.03
query42 0.04 0.02 0.03
query43 0.04 0.03 0.03
Total cold run time: 103.68 s
Total hot run time: 29.42 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix incorrect result when there are insert after delete for same key in one load [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #41701:
URL: https://github.com/apache/doris/pull/41701#discussion_r1796526991
##########
be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:
##########
@@ -558,6 +568,32 @@ Status VerticalSegmentWriter::_append_block_with_partial_content(RowsInBlock& da
return Status::OK();
}
+Status VerticalSegmentWriter::_filter_block_for_flexible_partial_update(
+ RowsInBlock& data, vectorized::MutableColumnPtr filter_column, int duplicate_rows,
+ std::string col_name) {
+ auto num_cols = data.block->columns();
+ auto* block = const_cast<vectorized::Block*>(data.block);
+ block->insert(
+ {std::move(filter_column), std::make_shared<vectorized::DataTypeUInt8>(), col_name});
+ RETURN_IF_ERROR(vectorized::Block::filter_block(block, num_cols, num_cols));
+ DCHECK_EQ(num_cols, data.block->columns());
+ int merged_rows = data.num_rows - block->rows();
+ VLOG_DEBUG << fmt::format(
+ "filter_block_for_flexible_partial_update[{}] after filter: "
+ "data.block:{}\n",
+ col_name, data.block->dump_data());
+ if (duplicate_rows != merged_rows) {
+ auto msg = fmt::format(
+ "filter_block_for_flexible_partial_update {}: duplicate_rows != merged_rows, "
+ "duplicate_keys={}, merged_rows={}, num_rows={}, mutable_block->rows()={}",
+ col_name, duplicate_rows, merged_rows, data.num_rows, block->rows());
+ DCHECK(false) << msg;
+ return Status::InternalError<false>(msg);
+ }
+ data.num_rows = block->rows();
+ return Status::OK();
+}
+
Status VerticalSegmentWriter::_append_block_with_flexible_partial_content(
Review Comment:
warning: function '_append_block_with_flexible_partial_content' has cognitive complexity of 86 (threshold 50) [readability-function-cognitive-complexity]
```cpp
Status VerticalSegmentWriter::_append_block_with_flexible_partial_content(
^
```
<details>
<summary>Additional context</summary>
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:614:** nesting level increased to 1
```cpp
auto get_skip_bitmaps = [&skip_bitmap_col_idx](const vectorized::Block* block) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:628:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
DBUG_EXECUTE_IF("VerticalSegmentWriter._append_block_with_flexible_partial_content.sleep",
^
```
**be/src/util/debug_points.h:36:** expanded from macro 'DBUG_EXECUTE_IF'
```cpp
if (UNLIKELY(config::enable_debug_points)) { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:628:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
DBUG_EXECUTE_IF("VerticalSegmentWriter._append_block_with_flexible_partial_content.sleep",
^
```
**be/src/util/debug_points.h:38:** expanded from macro 'DBUG_EXECUTE_IF'
```cpp
if (dp) { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:637:** nesting level increased to 1
```cpp
[&full_block, &data,
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:640:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
for (std::size_t cid {0}; cid < _num_sort_key_columns; cid++) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:642:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(_olap_data_convertor->set_source_content_with_specifid_column(
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:642:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
RETURN_IF_ERROR(_olap_data_convertor->set_source_content_with_specifid_column(
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:645:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (!status.ok()) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:653:** nesting level increased to 1
```cpp
auto encode_seq_column = [&data, &schema_has_sequence_col,
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:656:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (schema_has_sequence_col) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:658:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(_olap_data_convertor->set_source_content_with_specifid_column(
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:658:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
RETURN_IF_ERROR(_olap_data_convertor->set_source_content_with_specifid_column(
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:662:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (!status.ok()) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:673:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
RETURN_IF_ERROR(encode_key_columns(key_columns));
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:673:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(encode_key_columns(key_columns));
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:680:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
RETURN_IF_ERROR(encode_seq_column(seq_column));
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:680:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(encode_seq_column(seq_column));
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:687:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
if (schema_has_sequence_col) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:688:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(_merge_rows_for_sequence_column(data, skip_bitmaps, key_columns, seq_column,
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:688:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(_merge_rows_for_sequence_column(data, skip_bitmaps, key_columns, seq_column,
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:690:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (origin_rows != data.num_rows) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:693:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(encode_key_columns(key_columns));
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:693:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
RETURN_IF_ERROR(encode_key_columns(key_columns));
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:694:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(encode_seq_column(seq_column));
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:694:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
RETURN_IF_ERROR(encode_seq_column(seq_column));
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:705:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
RETURN_IF_ERROR(_merge_rows_for_insert_after_delete(data, skip_bitmaps, key_columns, seq_column,
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:705:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(_merge_rows_for_insert_after_delete(data, skip_bitmaps, key_columns, seq_column,
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:708:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
if (data.num_rows != origin_rows) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:711:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(encode_key_columns(key_columns));
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:711:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(encode_key_columns(key_columns));
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:712:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(encode_seq_column(seq_column));
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:712:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(encode_seq_column(seq_column));
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:719:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
for (std::size_t cid {0}; cid < _num_sort_key_columns; cid++) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:722:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(_column_writers[cid]->append(column->get_nullmap(), column->get_data(),
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:722:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(_column_writers[cid]->append(column->get_nullmap(), column->get_data(),
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:730:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
RETURN_IF_ERROR(_generate_flexible_read_plan(
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:730:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(_generate_flexible_read_plan(
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:736:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
if (config::enable_merge_on_write_correctness_check) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:742:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
RETURN_IF_ERROR(read_plan.fill_non_primary_key_columns(
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:742:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(read_plan.fill_non_primary_key_columns(
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:776:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
if (_num_rows_written != data.row_pos ||
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:785:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
RETURN_IF_ERROR(_generate_primary_key_index(_key_coders, key_columns, seq_column, data.num_rows,
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:785:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(_generate_primary_key_index(_key_coders, key_columns, seq_column, data.num_rows,
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix incorrect result when there are insert after delete for same key in one load [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2408476141
TeamCity be ut coverage result:
Function Coverage: 37.42% (9681/25871)
Line Coverage: 28.70% (80449/280325)
Region Coverage: 28.10% (41532/147820)
Branch Coverage: 24.68% (21115/85568)
Coverage Report: http://coverage.selectdb-in.cc/coverage/79fe26c74dbc3442b2b7b76045ec77054fcb9a10_79fe26c74dbc3442b2b7b76045ec77054fcb9a10/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix incorrect result when there are insert after delete for same key in one load [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #41701:
URL: https://github.com/apache/doris/pull/41701#discussion_r1801015444
##########
be/src/olap/memtable.cpp:
##########
@@ -578,6 +546,200 @@ void MemTable::_aggregate() {
}
}
+template <bool is_final>
+void MemTable::_aggregate_for_flexible_partial_update_without_seq_col(
+ const vectorized::ColumnsWithTypeAndName& block_data,
+ vectorized::MutableBlock& mutable_block, std::vector<RowInBlock*>& temp_row_in_blocks) {
+ RowInBlock* prev_row = nullptr;
+ int row_pos = -1;
+ auto& skip_bitmaps = assert_cast<vectorized::ColumnBitmap*>(
+ mutable_block.mutable_columns()[_skip_bitmap_col_idx].get())
+ ->get_data();
+ auto& delete_signs = assert_cast<vectorized::ColumnInt8*>(
+ mutable_block.mutable_columns()[_delete_sign_col_idx].get())
+ ->get_data();
+ RowInBlock* row_with_delete_sign {nullptr};
+ RowInBlock* row_without_delete_sign {nullptr};
+
+ auto finalize_rows = [&]() {
+ if (row_with_delete_sign != nullptr) {
+ temp_row_in_blocks.push_back(row_with_delete_sign);
+ _finalize_one_row<is_final, false>(row_with_delete_sign, block_data, ++row_pos);
+ row_with_delete_sign = nullptr;
+ }
+ if (row_without_delete_sign != nullptr) {
+ temp_row_in_blocks.push_back(row_without_delete_sign);
+ _finalize_one_row<is_final, false>(row_without_delete_sign, block_data, ++row_pos);
+ row_without_delete_sign = nullptr;
+ }
+ _arena->clear();
+ };
+
+ auto add_row = [&](RowInBlock* row, bool with_delete_sign) {
+ if (with_delete_sign) {
+ row_with_delete_sign = row;
+ } else {
+ row_without_delete_sign = row;
+ }
+ };
+ for (RowInBlock* cur_row : _row_in_blocks) {
+ const BitmapValue& skip_bitmap = skip_bitmaps[cur_row->_row_pos];
+ bool cur_row_has_delete_sign = (!skip_bitmap.contains(_delete_sign_col_unique_id) &&
+ delete_signs[cur_row->_row_pos] != 0);
+ prev_row =
+ (row_with_delete_sign == nullptr) ? row_without_delete_sign : row_with_delete_sign;
+ // compare keys, the keys of row_with_delete_sign and row_without_delete_sign is the same,
+ // choose any of them if it's valid
+ if (prev_row != nullptr && (*_vec_row_comparator)(prev_row, cur_row) == 0) {
+ if (cur_row_has_delete_sign) {
+ if (row_without_delete_sign != nullptr) {
+ // if there exits row without delete sign, remove it first
+ _clear_row_agg(row_without_delete_sign);
+ _stat.merged_rows++;
+ row_without_delete_sign = nullptr;
+ }
+ // and then unconditionally replace the previous row
+ prev_row = row_with_delete_sign;
+ } else {
+ prev_row = row_without_delete_sign;
+ }
+
+ if (prev_row == nullptr) {
+ add_row(cur_row, cur_row_has_delete_sign);
+ } else {
+ if (!prev_row->has_init_agg()) {
+ _init_row_for_agg(prev_row, mutable_block);
+ }
+ _stat.merged_rows++;
+ _aggregate_two_row_in_block<true>(mutable_block, cur_row, prev_row);
+ }
+ } else {
+ finalize_rows();
+ add_row(cur_row, cur_row_has_delete_sign);
+ }
+ }
+ // finalize the last lows
+ finalize_rows();
+}
+
+template <bool is_final>
+void MemTable::_aggregate_for_flexible_partial_update_with_seq_col(
Review Comment:
warning: function '_aggregate_for_flexible_partial_update_with_seq_col' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
void MemTable::_aggregate_for_flexible_partial_update_with_seq_col(
^
```
<details>
<summary>Additional context</summary>
**be/src/olap/memtable.cpp:625:** 113 lines including whitespace and comments (threshold 80)
```cpp
void MemTable::_aggregate_for_flexible_partial_update_with_seq_col(
^
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix incorrect result when there are insert after delete for same key in one load [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2413837341
<details>
<summary>TPC-H: <b>Total hot run time: 41508 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 99e53bd3da17aa9c7e310bf4b8d91beb7a81b396, data reload: false
------ Round 1 ----------------------------------
q1 17573 7585 7338 7338
q2 2039 160 158 158
q3 10565 1137 1273 1137
q4 10218 827 856 827
q5 7739 3100 3074 3074
q6 239 155 154 154
q7 1032 632 627 627
q8 9388 1972 1938 1938
q9 7146 6428 6435 6428
q10 7107 2400 2409 2400
q11 457 247 246 246
q12 532 224 236 224
q13 17815 3068 3032 3032
q14 240 216 212 212
q15 575 532 522 522
q16 682 586 583 583
q17 990 574 582 574
q18 7459 6796 6705 6705
q19 1365 995 1098 995
q20 473 187 194 187
q21 4034 3141 3279 3141
q22 1121 1035 1006 1006
Total cold run time: 108789 ms
Total hot run time: 41508 ms
----- Round 2, with runtime_filter_mode=off -----
q1 7240 7256 7262 7256
q2 332 225 232 225
q3 2966 2782 2808 2782
q4 1978 1709 1710 1709
q5 5506 5500 5507 5500
q6 225 142 144 142
q7 2137 1728 1732 1728
q8 3258 3411 3444 3411
q9 8573 8551 8554 8551
q10 3480 3442 3451 3442
q11 571 476 471 471
q12 768 602 588 588
q13 13234 3005 3038 3005
q14 295 253 258 253
q15 551 506 498 498
q16 676 646 643 643
q17 1844 1576 1531 1531
q18 7737 7399 7426 7399
q19 1670 1493 1451 1451
q20 2034 1813 1817 1813
q21 5474 5340 5243 5243
q22 1131 1012 992 992
Total cold run time: 71680 ms
Total hot run time: 58633 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2561942379
TeamCity be ut coverage result:
Function Coverage: 38.77% (10094/26035)
Line Coverage: 29.76% (85191/286298)
Region Coverage: 28.88% (43506/150664)
Branch Coverage: 25.41% (22174/87278)
Coverage Report: http://coverage.selectdb-in.cc/coverage/9cc0eab95240250a54bda6aa63dfd1cf1932c83e_9cc0eab95240250a54bda6aa63dfd1cf1932c83e/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix incorrect result when there are insert after delete for same key in one load [doris]
Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2451379167
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2490478643
<details>
<summary>ClickBench: <b>Total hot run time: 33.5 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2d818f829608fbd86f1492c0f3ea5cf6307e7494, data reload: false
query1 0.02 0.03 0.03
query2 0.08 0.05 0.04
query3 0.22 0.06 0.05
query4 1.66 0.08 0.09
query5 0.42 0.40 0.40
query6 1.16 0.67 0.67
query7 0.04 0.01 0.01
query8 0.05 0.04 0.04
query9 0.55 0.51 0.51
query10 0.55 0.55 0.56
query11 0.17 0.12 0.13
query12 0.16 0.13 0.13
query13 0.60 0.60 0.60
query14 2.73 2.85 2.80
query15 0.91 0.84 0.84
query16 0.37 0.38 0.38
query17 1.08 1.04 1.07
query18 0.20 0.20 0.20
query19 1.84 1.90 2.02
query20 0.01 0.01 0.01
query21 15.35 0.66 0.66
query22 4.50 7.25 1.88
query23 18.24 1.33 1.30
query24 2.08 0.23 0.23
query25 0.15 0.08 0.08
query26 0.27 0.18 0.19
query27 0.08 0.08 0.08
query28 13.21 1.18 1.16
query29 12.66 3.46 3.45
query30 0.24 0.06 0.06
query31 2.87 0.40 0.40
query32 3.25 0.49 0.49
query33 3.02 3.05 3.11
query34 16.76 4.55 4.47
query35 4.57 4.60 4.53
query36 0.69 0.48 0.48
query37 0.20 0.17 0.16
query38 0.17 0.15 0.14
query39 0.05 0.04 0.03
query40 0.16 0.14 0.13
query41 0.10 0.06 0.05
query42 0.07 0.05 0.05
query43 0.04 0.04 0.04
Total cold run time: 111.55 s
Total hot run time: 33.5 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2493253409
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix incorrect result when there are insert after delete for same key in one load [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #41701:
URL: https://github.com/apache/doris/pull/41701#discussion_r1806067141
##########
be/src/olap/partial_update_info.cpp:
##########
@@ -658,4 +666,336 @@
return Status::OK();
}
+BlockAggregator::BlockAggregator(segment_v2::VerticalSegmentWriter& vertical_segment_writer)
+ : _writer(vertical_segment_writer), _tablet_schema(*_writer._tablet_schema) {}
+
+Status BlockAggregator::aggregate_for_sequence_column(
+ vectorized::Block* block, size_t num_rows,
+ const std::vector<vectorized::IOlapColumnDataAccessor*>& key_columns,
+ vectorized::IOlapColumnDataAccessor* seq_column,
+ const std::vector<RowsetSharedPtr>& specified_rowsets,
+ std::vector<std::unique_ptr<SegmentCacheHandle>>& segment_caches) {
+ DCHECK_EQ(block->columns(), _tablet_schema.num_columns());
+ VLOG_DEBUG << fmt::format("BlockAggregator::aggregate_for_sequence_column enter: block:{}\n",
+ block->dump_data());
+ // the process logic here is the same as MemTable::_aggregate_for_flexible_partial_update_without_seq_col()
+ // after this function, there will be at most 2 rows for a specified key
+
+ std::vector<BitmapValue>* skip_bitmaps =
+ &(assert_cast<vectorized::ColumnBitmap*>(
+ block->get_by_position(_tablet_schema.skip_bitmap_col_idx())
+ .column->assume_mutable()
+ .get())
+ ->get_data());
+ const auto* delete_signs = BaseTablet::get_delete_sign_column_data(*block, num_rows);
+
+ auto seq_col_unique_id = _tablet_schema.column(_tablet_schema.sequence_col_idx()).unique_id();
+ auto delete_sign_col_unique_id =
+ _tablet_schema.column(_tablet_schema.delete_sign_idx()).unique_id();
+
+ auto filtered_block = _tablet_schema.create_block();
+ vectorized::MutableBlock output_block =
+ vectorized::MutableBlock::build_mutable_block(&filtered_block);
+ int cur_rows {0};
+ bool have_rows_with_delete_sign {false};
+
+ auto append_row = [&](int64_t rid, BitmapValue& skip_bitmap, bool have_delete_sign) {
+ if (have_delete_sign) {
+ // remove all the preious batched rows
+ if (cur_rows > 0) {
+ for (size_t cid {0}; cid < _tablet_schema.num_columns(); cid++) {
+ DCHECK_GE(output_block.mutable_columns()[cid]->size(), cur_rows);
+ output_block.mutable_columns()[cid]->pop_back(cur_rows);
+ }
+ }
+ output_block.add_row(block, rid);
+ cur_rows = 1;
+ have_rows_with_delete_sign = true;
+ VLOG_DEBUG << fmt::format(
+ "append a row with delete sign, after append, cur_rows={}, "
+ "output_block.rows()={}",
+ cur_rows, output_block.rows());
+ } else {
+ if ((cur_rows == 1 && !have_rows_with_delete_sign) || cur_rows == 2) {
+ for (size_t cid {0}; cid < _tablet_schema.num_columns(); cid++) {
+ if (cid == _tablet_schema.skip_bitmap_col_idx()) {
+ auto& cur_skip_bitmap = assert_cast<vectorized::ColumnBitmap*>(
+ output_block.mutable_columns()[cid].get())
+ ->get_data()
+ .back();
+ const auto& new_row_skip_bitmap =
+ assert_cast<vectorized::ColumnBitmap*>(
+ block->get_by_position(cid).column->assume_mutable().get())
+ ->get_data()[rid];
+ cur_skip_bitmap &= new_row_skip_bitmap;
+ continue;
+ }
+ if (!skip_bitmap.contains(_tablet_schema.column(cid).unique_id())) {
+ output_block.mutable_columns()[cid]->pop_back(1);
+ output_block.mutable_columns()[cid]->insert_from(
+ *block->get_by_position(cid).column, rid);
+ }
+ }
+ VLOG_DEBUG << fmt::format(
+ "merge a row, after merge, cur_rows={}, output_block.rows()={}", cur_rows,
+ output_block.rows());
+ } else {
+ output_block.add_row(block, rid);
+ cur_rows++;
+ VLOG_DEBUG << fmt::format(
+ "append a new row, after append, cur_rows={}, output_block.rows()={}",
+ cur_rows, output_block.rows());
+ }
+ }
+ };
+
+ // aggregate rows with same keys in block range [start, end)
+ auto aggregate_rows = [&](std::string key, int64_t start, int64_t end) -> Status {
+ VLOG_DEBUG << fmt::format("merge rows in range=({}-{})", start, end);
+ if (end - start == 1) {
+ output_block.add_row(block, start);
+ VLOG_DEBUG << fmt::format("append a row directly, rid={}", start);
+ return Status::OK();
+ }
+ cur_rows = 0;
+ have_rows_with_delete_sign = false;
+
+ RowLocation loc;
+ RowsetSharedPtr rowset;
+ std::string previous_encoded_seq_value {};
+ Status st = _writer._tablet->lookup_row_key(key, &_tablet_schema, false, specified_rowsets,
+ &loc, _writer._mow_context->max_version,
+ segment_caches, &rowset, true,
+ &previous_encoded_seq_value);
+ int64_t pos = start;
+ DCHECK(st.is<ErrorCode::KEY_NOT_FOUND>() || st.ok());
+
+ std::string cur_seq_val;
+ if (st.ok()) {
+ for (int64_t i {start}; i < end; i++) {
+ auto& skip_bitmap = skip_bitmaps->at(i);
+ bool row_has_sequence_col = (!skip_bitmap.contains(seq_col_unique_id));
+ // Find the first row that has larger sequence value than the existing row's
+ // or the first row that doesn't specify the sequence column.
+ // Discard all the rows before and begin to do aggregation from that row
+ if (!row_has_sequence_col) {
+ cur_seq_val = std::move(previous_encoded_seq_value);
+ pos = i;
+ break;
+ } else {
+ std::string seq_val {};
+ _writer._encode_seq_column(seq_column, i, &seq_val);
+ if (Slice {seq_val}.compare(Slice {previous_encoded_seq_value}) >= 0) {
+ cur_seq_val = std::move(seq_val);
+ pos = i;
+ break;
+ }
+ }
+ }
+ } else {
+ pos = start;
+ auto& skip_bitmap = skip_bitmaps->at(pos);
+ bool row_has_sequence_col = (!skip_bitmap.contains(seq_col_unique_id));
+ if (row_has_sequence_col) {
+ std::string seq_val {};
+ // for rows that don't specify seqeunce col, seq_val will be encoded to minial value
+ _writer._encode_seq_column(seq_column, pos, &seq_val);
+ cur_seq_val = std::move(seq_val);
+ } else {
+ cur_seq_val.clear();
+ RETURN_IF_ERROR(_writer._generate_encoded_default_seq_value(
+ _tablet_schema, *_writer._opts.rowset_ctx->partial_update_info,
+ &cur_seq_val));
+ }
+ }
+ cur_rows = 0;
+ VLOG_DEBUG << fmt::format("start pos={}, output_block.rows()={}", pos, output_block.rows());
+
+ for (int64_t rid {pos}; rid < end; rid++) {
+ auto& skip_bitmap = skip_bitmaps->at(rid);
+ bool row_has_sequence_col = (!skip_bitmap.contains(seq_col_unique_id));
+ bool have_delete_sign =
+ (!skip_bitmap.contains(delete_sign_col_unique_id) && delete_signs[rid] != 0);
+ if (!row_has_sequence_col) {
+ append_row(rid, skip_bitmap, have_delete_sign);
+ } else {
+ std::string seq_val {};
+ _writer._encode_seq_column(seq_column, rid, &seq_val);
+ if (Slice {seq_val}.compare(Slice {cur_seq_val}) >= 0) {
+ append_row(rid, skip_bitmap, have_delete_sign);
+ cur_seq_val = std::move(seq_val);
+ } else {
+ VLOG_DEBUG << fmt::format(
+ "skip rid={} becasue its seq value is lower than the previous", rid);
+ }
+ }
+ }
+ return Status::OK();
+ };
+
+ int same_key_rows {0};
+ std::string previous_key {};
+ for (size_t block_pos {0}; block_pos < num_rows; block_pos++) {
+ std::string key = _writer._full_encode_keys(key_columns, block_pos);
+ if (block_pos > 0 && previous_key == key) {
+ same_key_rows++;
+ } else {
+ if (same_key_rows > 0) {
+ RETURN_IF_ERROR(aggregate_rows(std::move(previous_key), block_pos - same_key_rows,
+ block_pos));
+ }
+ same_key_rows = 1;
+ }
+ previous_key = std::move(key);
+ }
+ if (same_key_rows > 0) {
+ RETURN_IF_ERROR(
+ aggregate_rows(std::move(previous_key), num_rows - same_key_rows, num_rows));
+ }
+
+ block->swap(output_block.to_block());
+ VLOG_DEBUG << fmt::format(
+ "BlockAggregator::aggregate_for_sequence_column after: data.block:{}\n",
+ block->dump_data());
+ return Status::OK();
+}
+
+Status BlockAggregator::aggregate_for_insert_after_delete(
Review Comment:
warning: function 'aggregate_for_insert_after_delete' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
Status BlockAggregator::aggregate_for_insert_after_delete(
^
```
<details>
<summary>Additional context</summary>
**be/src/olap/partial_update_info.cpp:862:** 109 lines including whitespace and comments (threshold 80)
```cpp
Status BlockAggregator::aggregate_for_insert_after_delete(
^
```
</details>
##########
be/src/olap/partial_update_info.cpp:
##########
@@ -658,4 +666,336 @@ Status FlexibleReadPlan::fill_non_primary_key_columns_for_row_store(
return Status::OK();
}
+BlockAggregator::BlockAggregator(segment_v2::VerticalSegmentWriter& vertical_segment_writer)
+ : _writer(vertical_segment_writer), _tablet_schema(*_writer._tablet_schema) {}
+
+Status BlockAggregator::aggregate_for_sequence_column(
Review Comment:
warning: function 'aggregate_for_sequence_column' has cognitive complexity of 97 (threshold 50) [readability-function-cognitive-complexity]
```cpp
Status BlockAggregator::aggregate_for_sequence_column(
^
```
<details>
<summary>Additional context</summary>
**be/src/olap/partial_update_info.cpp:701:** nesting level increased to 1
```cpp
auto append_row = [&](int64_t rid, BitmapValue& skip_bitmap, bool have_delete_sign) {
^
```
**be/src/olap/partial_update_info.cpp:702:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (have_delete_sign) {
^
```
**be/src/olap/partial_update_info.cpp:704:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (cur_rows > 0) {
^
```
**be/src/olap/partial_update_info.cpp:705:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
for (size_t cid {0}; cid < _tablet_schema.num_columns(); cid++) {
^
```
**be/src/olap/partial_update_info.cpp:717:** +1, nesting level increased to 2
```cpp
} else {
^
```
**be/src/olap/partial_update_info.cpp:718:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if ((cur_rows == 1 && !have_rows_with_delete_sign) || cur_rows == 2) {
^
```
**be/src/olap/partial_update_info.cpp:718:** +1
```cpp
if ((cur_rows == 1 && !have_rows_with_delete_sign) || cur_rows == 2) {
^
```
**be/src/olap/partial_update_info.cpp:718:** +1
```cpp
if ((cur_rows == 1 && !have_rows_with_delete_sign) || cur_rows == 2) {
^
```
**be/src/olap/partial_update_info.cpp:719:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
for (size_t cid {0}; cid < _tablet_schema.num_columns(); cid++) {
^
```
**be/src/olap/partial_update_info.cpp:720:** +5, including nesting penalty of 4, nesting level increased to 5
```cpp
if (cid == _tablet_schema.skip_bitmap_col_idx()) {
^
```
**be/src/olap/partial_update_info.cpp:732:** +5, including nesting penalty of 4, nesting level increased to 5
```cpp
if (!skip_bitmap.contains(_tablet_schema.column(cid).unique_id())) {
^
```
**be/src/olap/partial_update_info.cpp:741:** +1, nesting level increased to 3
```cpp
} else {
^
```
**be/src/olap/partial_update_info.cpp:752:** nesting level increased to 1
```cpp
auto aggregate_rows = [&](std::string key, int64_t start, int64_t end) -> Status {
^
```
**be/src/olap/partial_update_info.cpp:754:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (end - start == 1) {
^
```
**be/src/olap/partial_update_info.cpp:770:** +1
```cpp
DCHECK(st.is<ErrorCode::KEY_NOT_FOUND>() || st.ok());
^
```
**be/src/olap/partial_update_info.cpp:773:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (st.ok()) {
^
```
**be/src/olap/partial_update_info.cpp:774:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
for (int64_t i {start}; i < end; i++) {
^
```
**be/src/olap/partial_update_info.cpp:780:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
if (!row_has_sequence_col) {
^
```
**be/src/olap/partial_update_info.cpp:784:** +1, nesting level increased to 4
```cpp
} else {
^
```
**be/src/olap/partial_update_info.cpp:787:** +5, including nesting penalty of 4, nesting level increased to 5
```cpp
if (Slice {seq_val}.compare(Slice {previous_encoded_seq_value}) >= 0) {
^
```
**be/src/olap/partial_update_info.cpp:794:** +1, nesting level increased to 2
```cpp
} else {
^
```
**be/src/olap/partial_update_info.cpp:798:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (row_has_sequence_col) {
^
```
**be/src/olap/partial_update_info.cpp:803:** +1, nesting level increased to 3
```cpp
} else {
^
```
**be/src/olap/partial_update_info.cpp:805:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
RETURN_IF_ERROR(_writer._generate_encoded_default_seq_value(
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/partial_update_info.cpp:805:** +5, including nesting penalty of 4, nesting level increased to 5
```cpp
RETURN_IF_ERROR(_writer._generate_encoded_default_seq_value(
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/partial_update_info.cpp:813:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
for (int64_t rid {pos}; rid < end; rid++) {
^
```
**be/src/olap/partial_update_info.cpp:817:** +1
```cpp
(!skip_bitmap.contains(delete_sign_col_unique_id) && delete_signs[rid] != 0);
^
```
**be/src/olap/partial_update_info.cpp:818:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (!row_has_sequence_col) {
^
```
**be/src/olap/partial_update_info.cpp:820:** +1, nesting level increased to 3
```cpp
} else {
^
```
**be/src/olap/partial_update_info.cpp:823:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
if (Slice {seq_val}.compare(Slice {cur_seq_val}) >= 0) {
^
```
**be/src/olap/partial_update_info.cpp:826:** +1, nesting level increased to 4
```cpp
} else {
^
```
**be/src/olap/partial_update_info.cpp:837:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
for (size_t block_pos {0}; block_pos < num_rows; block_pos++) {
^
```
**be/src/olap/partial_update_info.cpp:839:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (block_pos > 0 && previous_key == key) {
^
```
**be/src/olap/partial_update_info.cpp:839:** +1
```cpp
if (block_pos > 0 && previous_key == key) {
^
```
**be/src/olap/partial_update_info.cpp:841:** +1, nesting level increased to 2
```cpp
} else {
^
```
**be/src/olap/partial_update_info.cpp:842:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (same_key_rows > 0) {
^
```
**be/src/olap/partial_update_info.cpp:843:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
RETURN_IF_ERROR(aggregate_rows(std::move(previous_key), block_pos - same_key_rows,
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/partial_update_info.cpp:843:** +5, including nesting penalty of 4, nesting level increased to 5
```cpp
RETURN_IF_ERROR(aggregate_rows(std::move(previous_key), block_pos - same_key_rows,
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/partial_update_info.cpp:850:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
if (same_key_rows > 0) {
^
```
**be/src/olap/partial_update_info.cpp:851:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/partial_update_info.cpp:851:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
</details>
##########
be/src/olap/partial_update_info.cpp:
##########
@@ -658,4 +666,336 @@
return Status::OK();
}
+BlockAggregator::BlockAggregator(segment_v2::VerticalSegmentWriter& vertical_segment_writer)
+ : _writer(vertical_segment_writer), _tablet_schema(*_writer._tablet_schema) {}
+
+Status BlockAggregator::aggregate_for_sequence_column(
Review Comment:
warning: function 'aggregate_for_sequence_column' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
Status BlockAggregator::aggregate_for_sequence_column(
^
```
<details>
<summary>Additional context</summary>
**be/src/olap/partial_update_info.cpp:671:** 184 lines including whitespace and comments (threshold 80)
```cpp
Status BlockAggregator::aggregate_for_sequence_column(
^
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix incorrect result when there are insert after delete for same key in one load [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #41701:
URL: https://github.com/apache/doris/pull/41701#discussion_r1806348726
##########
be/src/olap/partial_update_info.cpp:
##########
@@ -658,4 +664,422 @@ Status FlexibleReadPlan::fill_non_primary_key_columns_for_row_store(
return Status::OK();
}
+BlockAggregator::BlockAggregator(segment_v2::VerticalSegmentWriter& vertical_segment_writer)
+ : _writer(vertical_segment_writer), _tablet_schema(*_writer._tablet_schema) {}
+
+void BlockAggregator::merge_one_row(vectorized::MutableBlock& dst_block,
+ vectorized::Block* src_block, int64_t rid,
+ BitmapValue& skip_bitmap) {
+ for (size_t cid {_tablet_schema.num_key_columns()}; cid < _tablet_schema.num_columns(); cid++) {
+ if (cid == _tablet_schema.skip_bitmap_col_idx()) {
+ auto& cur_skip_bitmap =
+ assert_cast<vectorized::ColumnBitmap*>(dst_block.mutable_columns()[cid].get())
+ ->get_data()
+ .back();
+ const auto& new_row_skip_bitmap =
+ assert_cast<vectorized::ColumnBitmap*>(
+ src_block->get_by_position(cid).column->assume_mutable().get())
+ ->get_data()[rid];
+ cur_skip_bitmap &= new_row_skip_bitmap;
+ continue;
+ }
+ if (!skip_bitmap.contains(_tablet_schema.column(cid).unique_id())) {
+ dst_block.mutable_columns()[cid]->pop_back(1);
+ dst_block.mutable_columns()[cid]->insert_from(*src_block->get_by_position(cid).column,
+ rid);
+ }
+ }
+ VLOG_DEBUG << fmt::format("merge a row, after merge, output_block.rows()={}, state: {}",
+ dst_block.rows(), _state.to_string());
+}
+
+void BlockAggregator::append_one_row(vectorized::MutableBlock& dst_block,
+ vectorized::Block* src_block, int64_t rid) {
+ dst_block.add_row(src_block, rid);
+ _state.rows++;
+ VLOG_DEBUG << fmt::format("append a new row, after append, output_block.rows()={}, state: {}",
+ dst_block.rows(), _state.to_string());
+}
+
+void BlockAggregator::remove_last_n_rows(vectorized::MutableBlock& dst_block, int n) {
+ if (n > 0) {
+ for (size_t cid {0}; cid < _tablet_schema.num_columns(); cid++) {
+ DCHECK_GE(dst_block.mutable_columns()[cid]->size(), n);
+ dst_block.mutable_columns()[cid]->pop_back(n);
+ }
+ }
+}
+
+void BlockAggregator::append_or_merge_row(vectorized::MutableBlock& dst_block,
+ vectorized::Block* src_block, int64_t rid,
+ BitmapValue& skip_bitmap, bool have_delete_sign) {
+ if (have_delete_sign) {
+ // remove all the previous batched rows
+ remove_last_n_rows(dst_block, _state.rows);
+ _state.rows = 0;
+ _state.has_row_with_delete_sign = true;
+
+ append_one_row(dst_block, src_block, rid);
+ } else {
+ if (_state.should_merge()) {
+ merge_one_row(dst_block, src_block, rid, skip_bitmap);
+ } else {
+ append_one_row(dst_block, src_block, rid);
+ }
+ }
+};
+
+Status BlockAggregator::aggregate_rows(
+ vectorized::MutableBlock& output_block, vectorized::Block* block, int64_t start,
+ int64_t end, std::string key, std::vector<BitmapValue>* skip_bitmaps,
Review Comment:
warning: function 'aggregate_rows' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
Status BlockAggregator::aggregate_rows(
^
```
<details>
<summary>Additional context</summary>
**be/src/olap/partial_update_info.cpp:733:** 82 lines including whitespace and comments (threshold 80)
```cpp
Status BlockAggregator::aggregate_rows(
^
```
</details>
##########
be/src/olap/partial_update_info.cpp:
##########
@@ -658,4 +664,422 @@
return Status::OK();
}
+BlockAggregator::BlockAggregator(segment_v2::VerticalSegmentWriter& vertical_segment_writer)
+ : _writer(vertical_segment_writer), _tablet_schema(*_writer._tablet_schema) {}
+
+void BlockAggregator::merge_one_row(vectorized::MutableBlock& dst_block,
+ vectorized::Block* src_block, int64_t rid,
+ BitmapValue& skip_bitmap) {
+ for (size_t cid {_tablet_schema.num_key_columns()}; cid < _tablet_schema.num_columns(); cid++) {
+ if (cid == _tablet_schema.skip_bitmap_col_idx()) {
+ auto& cur_skip_bitmap =
+ assert_cast<vectorized::ColumnBitmap*>(dst_block.mutable_columns()[cid].get())
+ ->get_data()
+ .back();
+ const auto& new_row_skip_bitmap =
+ assert_cast<vectorized::ColumnBitmap*>(
+ src_block->get_by_position(cid).column->assume_mutable().get())
+ ->get_data()[rid];
+ cur_skip_bitmap &= new_row_skip_bitmap;
+ continue;
+ }
+ if (!skip_bitmap.contains(_tablet_schema.column(cid).unique_id())) {
+ dst_block.mutable_columns()[cid]->pop_back(1);
+ dst_block.mutable_columns()[cid]->insert_from(*src_block->get_by_position(cid).column,
+ rid);
+ }
+ }
+ VLOG_DEBUG << fmt::format("merge a row, after merge, output_block.rows()={}, state: {}",
+ dst_block.rows(), _state.to_string());
+}
+
+void BlockAggregator::append_one_row(vectorized::MutableBlock& dst_block,
+ vectorized::Block* src_block, int64_t rid) {
+ dst_block.add_row(src_block, rid);
+ _state.rows++;
+ VLOG_DEBUG << fmt::format("append a new row, after append, output_block.rows()={}, state: {}",
+ dst_block.rows(), _state.to_string());
+}
+
+void BlockAggregator::remove_last_n_rows(vectorized::MutableBlock& dst_block, int n) {
+ if (n > 0) {
+ for (size_t cid {0}; cid < _tablet_schema.num_columns(); cid++) {
+ DCHECK_GE(dst_block.mutable_columns()[cid]->size(), n);
+ dst_block.mutable_columns()[cid]->pop_back(n);
+ }
+ }
+}
+
+void BlockAggregator::append_or_merge_row(vectorized::MutableBlock& dst_block,
+ vectorized::Block* src_block, int64_t rid,
+ BitmapValue& skip_bitmap, bool have_delete_sign) {
+ if (have_delete_sign) {
+ // remove all the previous batched rows
+ remove_last_n_rows(dst_block, _state.rows);
+ _state.rows = 0;
+ _state.has_row_with_delete_sign = true;
+
+ append_one_row(dst_block, src_block, rid);
+ } else {
+ if (_state.should_merge()) {
+ merge_one_row(dst_block, src_block, rid, skip_bitmap);
+ } else {
+ append_one_row(dst_block, src_block, rid);
+ }
+ }
+};
+
+Status BlockAggregator::aggregate_rows(
+ vectorized::MutableBlock& output_block, vectorized::Block* block, int64_t start,
+ int64_t end, std::string key, std::vector<BitmapValue>* skip_bitmaps,
+ const signed char* delete_signs, vectorized::IOlapColumnDataAccessor* seq_column,
+ const std::vector<RowsetSharedPtr>& specified_rowsets,
+ std::vector<std::unique_ptr<SegmentCacheHandle>>& segment_caches) {
+ VLOG_DEBUG << fmt::format("merge rows in range=({}-{})", start, end);
+ if (end - start == 1) {
+ output_block.add_row(block, start);
+ VLOG_DEBUG << fmt::format("append a row directly, rid={}", start);
+ return Status::OK();
+ }
+
+ auto seq_col_unique_id = _tablet_schema.column(_tablet_schema.sequence_col_idx()).unique_id();
+ auto delete_sign_col_unique_id =
+ _tablet_schema.column(_tablet_schema.delete_sign_idx()).unique_id();
+
+ _state.reset();
+
+ RowLocation loc;
+ RowsetSharedPtr rowset;
+ std::string previous_encoded_seq_value {};
+ Status st = _writer._tablet->lookup_row_key(
+ key, &_tablet_schema, false, specified_rowsets, &loc, _writer._mow_context->max_version,
+ segment_caches, &rowset, true, &previous_encoded_seq_value);
+ int64_t pos = start;
+ DCHECK(st.is<ErrorCode::KEY_NOT_FOUND>() || st.ok());
+
+ std::string cur_seq_val;
+ if (st.ok()) {
+ for (int64_t i {start}; i < end; i++) {
+ auto& skip_bitmap = skip_bitmaps->at(i);
+ bool row_has_sequence_col = (!skip_bitmap.contains(seq_col_unique_id));
+ // Find the first row that has larger sequence value than the existing row's
+ // or the first row that doesn't specify the sequence column.
+ // Discard all the rows before and begin to do aggregation from that row
+ if (!row_has_sequence_col) {
+ cur_seq_val = std::move(previous_encoded_seq_value);
+ pos = i;
+ break;
+ } else {
+ std::string seq_val {};
+ _writer._encode_seq_column(seq_column, i, &seq_val);
+ if (Slice {seq_val}.compare(Slice {previous_encoded_seq_value}) >= 0) {
+ cur_seq_val = std::move(seq_val);
+ pos = i;
+ break;
+ }
+ }
+ }
+ } else {
+ pos = start;
+ auto& skip_bitmap = skip_bitmaps->at(pos);
+ bool row_has_sequence_col = (!skip_bitmap.contains(seq_col_unique_id));
+ if (row_has_sequence_col) {
+ std::string seq_val {};
+ // for rows that don't specify seqeunce col, seq_val will be encoded to minial value
+ _writer._encode_seq_column(seq_column, pos, &seq_val);
+ cur_seq_val = std::move(seq_val);
+ } else {
+ cur_seq_val.clear();
+ RETURN_IF_ERROR(_writer._generate_encoded_default_seq_value(
+ _tablet_schema, *_writer._opts.rowset_ctx->partial_update_info, &cur_seq_val));
+ }
+ }
+ VLOG_DEBUG << fmt::format("start pos={}, output_block.rows()={}", pos, output_block.rows());
+
+ for (int64_t rid {pos}; rid < end; rid++) {
+ auto& skip_bitmap = skip_bitmaps->at(rid);
+ bool row_has_sequence_col = (!skip_bitmap.contains(seq_col_unique_id));
+ bool have_delete_sign =
+ (!skip_bitmap.contains(delete_sign_col_unique_id) && delete_signs[rid] != 0);
+ if (!row_has_sequence_col) {
+ append_or_merge_row(output_block, block, rid, skip_bitmap, have_delete_sign);
+ } else {
+ std::string seq_val {};
+ _writer._encode_seq_column(seq_column, rid, &seq_val);
+ if (Slice {seq_val}.compare(Slice {cur_seq_val}) >= 0) {
+ append_or_merge_row(output_block, block, rid, skip_bitmap, have_delete_sign);
+ cur_seq_val = std::move(seq_val);
+ } else {
+ VLOG_DEBUG << fmt::format(
+ "skip rid={} becasue its seq value is lower than the previous", rid);
+ }
+ }
+ }
+ return Status::OK();
+};
+
+Status BlockAggregator::aggregate_for_sequence_column(
+ vectorized::Block* block, size_t num_rows,
+ const std::vector<vectorized::IOlapColumnDataAccessor*>& key_columns,
+ vectorized::IOlapColumnDataAccessor* seq_column,
+ const std::vector<RowsetSharedPtr>& specified_rowsets,
+ std::vector<std::unique_ptr<SegmentCacheHandle>>& segment_caches) {
+ DCHECK_EQ(block->columns(), _tablet_schema.num_columns());
+ VLOG_DEBUG << fmt::format("BlockAggregator::aggregate_for_sequence_column enter: block:{}\n",
+ block->dump_data());
+ // the process logic here is the same as MemTable::_aggregate_for_flexible_partial_update_without_seq_col()
+ // after this function, there will be at most 2 rows for a specified key
+ std::vector<BitmapValue>* skip_bitmaps =
+ &(assert_cast<vectorized::ColumnBitmap*>(
+ block->get_by_position(_tablet_schema.skip_bitmap_col_idx())
+ .column->assume_mutable()
+ .get())
+ ->get_data());
+ const auto* delete_signs = BaseTablet::get_delete_sign_column_data(*block, num_rows);
+
+ auto filtered_block = _tablet_schema.create_block();
+ vectorized::MutableBlock output_block =
+ vectorized::MutableBlock::build_mutable_block(&filtered_block);
+
+ int same_key_rows {0};
+ std::string previous_key {};
+ for (size_t block_pos {0}; block_pos < num_rows; block_pos++) {
+ std::string key = _writer._full_encode_keys(key_columns, block_pos);
+ if (block_pos > 0 && previous_key == key) {
+ same_key_rows++;
+ } else {
+ if (same_key_rows > 0) {
+ RETURN_IF_ERROR(aggregate_rows(output_block, block, block_pos - same_key_rows,
+ block_pos, std::move(previous_key), skip_bitmaps,
+ delete_signs, seq_column, specified_rowsets,
+ segment_caches));
+ }
+ same_key_rows = 1;
+ }
+ previous_key = std::move(key);
+ }
+ if (same_key_rows > 0) {
+ RETURN_IF_ERROR(aggregate_rows(output_block, block, num_rows - same_key_rows, num_rows,
+ std::move(previous_key), skip_bitmaps, delete_signs,
+ seq_column, specified_rowsets, segment_caches));
+ }
+
+ block->swap(output_block.to_block());
+ VLOG_DEBUG << fmt::format(
+ "BlockAggregator::aggregate_for_sequence_column after: data.block:{}\n",
+ block->dump_data());
+ return Status::OK();
+}
+
+Status BlockAggregator::fill_sequence_column(vectorized::Block* block, size_t num_rows,
+ const FixedReadPlan& read_plan,
+ std::vector<BitmapValue>& skip_bitmaps) {
+ DCHECK(_tablet_schema.has_sequence_col());
+ std::vector<uint32_t> cids {static_cast<uint32_t>(_tablet_schema.sequence_col_idx())};
+ auto seq_col_unique_id = _tablet_schema.column(_tablet_schema.sequence_col_idx()).unique_id();
+
+ auto seq_col_block = _tablet_schema.create_block_by_cids(cids);
+ auto tmp_block = _tablet_schema.create_block_by_cids(cids);
+ std::map<uint32_t, uint32_t> read_index;
+ RETURN_IF_ERROR(read_plan.read_columns_by_plan(_tablet_schema, cids, _writer._rsid_to_rowset,
+ seq_col_block, &read_index));
+
+ auto new_seq_col_ptr = tmp_block.get_by_position(0).column->assume_mutable();
+ const auto& old_seq_col_ptr = *seq_col_block.get_by_position(0).column;
+ const auto& cur_seq_col_ptr = *block->get_by_position(_tablet_schema.sequence_col_idx()).column;
+ for (size_t block_pos {0}; block_pos < num_rows; block_pos++) {
+ if (read_index.contains(block_pos)) {
+ new_seq_col_ptr->insert_from(old_seq_col_ptr, read_index[block_pos]);
+ skip_bitmaps[block_pos].remove(seq_col_unique_id);
+ } else {
+ new_seq_col_ptr->insert_from(cur_seq_col_ptr, block_pos);
+ }
+ }
+ block->replace_by_position(_tablet_schema.sequence_col_idx(), std::move(new_seq_col_ptr));
+ return Status::OK();
+}
+
+Status BlockAggregator::aggregate_for_insert_after_delete(
+ vectorized::Block* block, size_t num_rows,
+ const std::vector<vectorized::IOlapColumnDataAccessor*>& key_columns,
Review Comment:
warning: function 'aggregate_for_insert_after_delete' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
Status BlockAggregator::aggregate_for_insert_after_delete(
^
```
<details>
<summary>Additional context</summary>
**be/src/olap/partial_update_info.cpp:903:** 89 lines including whitespace and comments (threshold 80)
```cpp
Status BlockAggregator::aggregate_for_insert_after_delete(
^
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix incorrect result when there are insert after delete for same key in one load [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #41701:
URL: https://github.com/apache/doris/pull/41701#discussion_r1806343832
##########
be/src/olap/partial_update_info.cpp:
##########
@@ -658,4 +666,422 @@ Status FlexibleReadPlan::fill_non_primary_key_columns_for_row_store(
return Status::OK();
}
+BlockAggregator::BlockAggregator(segment_v2::VerticalSegmentWriter& vertical_segment_writer)
+ : _writer(vertical_segment_writer), _tablet_schema(*_writer._tablet_schema) {}
+
+void BlockAggregator::merge_one_row(vectorized::MutableBlock& dst_block,
+ vectorized::Block* src_block, int64_t rid,
+ BitmapValue& skip_bitmap) {
+ for (size_t cid {_tablet_schema.num_key_columns()}; cid < _tablet_schema.num_columns(); cid++) {
+ if (cid == _tablet_schema.skip_bitmap_col_idx()) {
+ auto& cur_skip_bitmap =
+ assert_cast<vectorized::ColumnBitmap*>(dst_block.mutable_columns()[cid].get())
+ ->get_data()
+ .back();
+ const auto& new_row_skip_bitmap =
+ assert_cast<vectorized::ColumnBitmap*>(
+ src_block->get_by_position(cid).column->assume_mutable().get())
+ ->get_data()[rid];
+ cur_skip_bitmap &= new_row_skip_bitmap;
+ continue;
+ }
+ if (!skip_bitmap.contains(_tablet_schema.column(cid).unique_id())) {
+ dst_block.mutable_columns()[cid]->pop_back(1);
+ dst_block.mutable_columns()[cid]->insert_from(*src_block->get_by_position(cid).column,
+ rid);
+ }
+ }
+ VLOG_DEBUG << fmt::format("merge a row, after merge, output_block.rows()={}, state: {}",
+ dst_block.rows(), _state.to_string());
+}
+
+void BlockAggregator::append_one_row(vectorized::MutableBlock& dst_block,
+ vectorized::Block* src_block, int64_t rid) {
+ dst_block.add_row(src_block, rid);
+ _state.rows++;
+ VLOG_DEBUG << fmt::format("append a new row, after append, output_block.rows()={}, state: {}",
+ dst_block.rows(), _state.to_string());
+}
+
+void BlockAggregator::remove_last_n_rows(vectorized::MutableBlock& dst_block, int n) {
+ if (n > 0) {
+ for (size_t cid {0}; cid < _tablet_schema.num_columns(); cid++) {
+ DCHECK_GE(dst_block.mutable_columns()[cid]->size(), n);
+ dst_block.mutable_columns()[cid]->pop_back(n);
+ }
+ }
+}
+
+void BlockAggregator::append_or_merge_row(vectorized::MutableBlock& dst_block,
+ vectorized::Block* src_block, int64_t rid,
+ BitmapValue& skip_bitmap, bool have_delete_sign) {
+ if (have_delete_sign) {
+ // remove all the previous batched rows
+ remove_rows(dst_block, _state.rows);
+ _state.rows = 0;
+ _state.has_row_with_delete_sign = true;
+
+ append_one_row(dst_block, src_block, rid);
+ } else {
+ if (_state.should_merge()) {
+ merge_one_row(dst_block, src_block, rid, skip_bitmap);
+ } else {
+ append_one_row(dst_block, src_block, rid);
+ }
+ }
+};
+
+Status BlockAggregator::aggregate_rows(
+ vectorized::MutableBlock& output_block, vectorized::Block* block, int64_t start,
+ int64_t end, std::string key, std::vector<BitmapValue>* skip_bitmaps,
+ const signed char* delete_signs, vectorized::IOlapColumnDataAccessor* seq_column,
+ const std::vector<RowsetSharedPtr>& specified_rowsets,
+ std::vector<std::unique_ptr<SegmentCacheHandle>>& segment_caches) {
+ VLOG_DEBUG << fmt::format("merge rows in range=({}-{})", start, end);
+ if (end - start == 1) {
+ output_block.add_row(block, start);
+ VLOG_DEBUG << fmt::format("append a row directly, rid={}", start);
+ return Status::OK();
+ }
+
+ auto seq_col_unique_id = _tablet_schema.column(_tablet_schema.sequence_col_idx()).unique_id();
+ auto delete_sign_col_unique_id =
+ _tablet_schema.column(_tablet_schema.delete_sign_idx()).unique_id();
+
+ _state.reset();
+
+ RowLocation loc;
+ RowsetSharedPtr rowset;
+ std::string previous_encoded_seq_value {};
+ Status st = _writer._tablet->lookup_row_key(
+ key, &_tablet_schema, false, specified_rowsets, &loc, _writer._mow_context->max_version,
+ segment_caches, &rowset, true, &previous_encoded_seq_value);
+ int64_t pos = start;
+ DCHECK(st.is<ErrorCode::KEY_NOT_FOUND>() || st.ok());
+
+ std::string cur_seq_val;
+ if (st.ok()) {
+ for (int64_t i {start}; i < end; i++) {
+ auto& skip_bitmap = skip_bitmaps->at(i);
+ bool row_has_sequence_col = (!skip_bitmap.contains(seq_col_unique_id));
+ // Find the first row that has larger sequence value than the existing row's
+ // or the first row that doesn't specify the sequence column.
+ // Discard all the rows before and begin to do aggregation from that row
+ if (!row_has_sequence_col) {
+ cur_seq_val = std::move(previous_encoded_seq_value);
+ pos = i;
+ break;
+ } else {
+ std::string seq_val {};
+ _writer._encode_seq_column(seq_column, i, &seq_val);
+ if (Slice {seq_val}.compare(Slice {previous_encoded_seq_value}) >= 0) {
+ cur_seq_val = std::move(seq_val);
+ pos = i;
+ break;
+ }
+ }
+ }
+ } else {
+ pos = start;
+ auto& skip_bitmap = skip_bitmaps->at(pos);
+ bool row_has_sequence_col = (!skip_bitmap.contains(seq_col_unique_id));
+ if (row_has_sequence_col) {
+ std::string seq_val {};
+ // for rows that don't specify seqeunce col, seq_val will be encoded to minial value
+ _writer._encode_seq_column(seq_column, pos, &seq_val);
+ cur_seq_val = std::move(seq_val);
+ } else {
+ cur_seq_val.clear();
+ RETURN_IF_ERROR(_writer._generate_encoded_default_seq_value(
+ _tablet_schema, *_writer._opts.rowset_ctx->partial_update_info, &cur_seq_val));
+ }
+ }
+ VLOG_DEBUG << fmt::format("start pos={}, output_block.rows()={}", pos, output_block.rows());
+
+ for (int64_t rid {pos}; rid < end; rid++) {
+ auto& skip_bitmap = skip_bitmaps->at(rid);
+ bool row_has_sequence_col = (!skip_bitmap.contains(seq_col_unique_id));
+ bool have_delete_sign =
+ (!skip_bitmap.contains(delete_sign_col_unique_id) && delete_signs[rid] != 0);
+ if (!row_has_sequence_col) {
+ append_or_merge_row(output_block, block, rid, skip_bitmap, have_delete_sign);
+ } else {
+ std::string seq_val {};
+ _writer._encode_seq_column(seq_column, rid, &seq_val);
+ if (Slice {seq_val}.compare(Slice {cur_seq_val}) >= 0) {
+ append_or_merge_row(output_block, block, rid, skip_bitmap, have_delete_sign);
+ cur_seq_val = std::move(seq_val);
+ } else {
+ VLOG_DEBUG << fmt::format(
+ "skip rid={} becasue its seq value is lower than the previous", rid);
+ }
+ }
+ }
+ return Status::OK();
+};
+
+Status BlockAggregator::aggregate_for_sequence_column(
+ vectorized::Block* block, size_t num_rows,
+ const std::vector<vectorized::IOlapColumnDataAccessor*>& key_columns,
+ vectorized::IOlapColumnDataAccessor* seq_column,
+ const std::vector<RowsetSharedPtr>& specified_rowsets,
+ std::vector<std::unique_ptr<SegmentCacheHandle>>& segment_caches) {
+ DCHECK_EQ(block->columns(), _tablet_schema.num_columns());
+ VLOG_DEBUG << fmt::format("BlockAggregator::aggregate_for_sequence_column enter: block:{}\n",
+ block->dump_data());
+ // the process logic here is the same as MemTable::_aggregate_for_flexible_partial_update_without_seq_col()
+ // after this function, there will be at most 2 rows for a specified key
+ std::vector<BitmapValue>* skip_bitmaps =
+ &(assert_cast<vectorized::ColumnBitmap*>(
+ block->get_by_position(_tablet_schema.skip_bitmap_col_idx())
+ .column->assume_mutable()
+ .get())
+ ->get_data());
+ const auto* delete_signs = BaseTablet::get_delete_sign_column_data(*block, num_rows);
+
+ auto filtered_block = _tablet_schema.create_block();
+ vectorized::MutableBlock output_block =
+ vectorized::MutableBlock::build_mutable_block(&filtered_block);
+
+ int same_key_rows {0};
+ std::string previous_key {};
+ for (size_t block_pos {0}; block_pos < num_rows; block_pos++) {
+ std::string key = _writer._full_encode_keys(key_columns, block_pos);
+ if (block_pos > 0 && previous_key == key) {
+ same_key_rows++;
+ } else {
+ if (same_key_rows > 0) {
+ RETURN_IF_ERROR(aggregate_rows(output_block, block, block_pos - same_key_rows,
+ block_pos, std::move(previous_key), skip_bitmaps,
+ delete_signs, seq_column, specified_rowsets,
+ segment_caches));
+ }
+ same_key_rows = 1;
+ }
+ previous_key = std::move(key);
+ }
+ if (same_key_rows > 0) {
+ RETURN_IF_ERROR(aggregate_rows(output_block, block, num_rows - same_key_rows, num_rows,
+ std::move(previous_key), skip_bitmaps, delete_signs,
+ seq_column, specified_rowsets, segment_caches));
+ }
+
+ block->swap(output_block.to_block());
+ VLOG_DEBUG << fmt::format(
+ "BlockAggregator::aggregate_for_sequence_column after: data.block:{}\n",
+ block->dump_data());
+ return Status::OK();
+}
+
+Status BlockAggregator::fill_sequence_column(vectorized::Block* block, size_t num_rows,
+ const FixedReadPlan& read_plan,
+ std::vector<BitmapValue>& skip_bitmaps) {
+ DCHECK(_tablet_schema.has_sequence_col());
+ std::vector<uint32_t> cids {static_cast<uint32_t>(_tablet_schema.sequence_col_idx())};
+ auto seq_col_unique_id = _tablet_schema.column(_tablet_schema.sequence_col_idx()).unique_id();
+
+ auto seq_col_block = _tablet_schema.create_block_by_cids(cids);
+ auto tmp_block = _tablet_schema.create_block_by_cids(cids);
+ std::map<uint32_t, uint32_t> read_index;
+ RETURN_IF_ERROR(read_plan.read_columns_by_plan(_tablet_schema, cids, _writer._rsid_to_rowset,
+ seq_col_block, &read_index));
+
+ auto new_seq_col_ptr = tmp_block.get_by_position(0).column->assume_mutable();
+ const auto& old_seq_col_ptr = *seq_col_block.get_by_position(0).column;
+ const auto& cur_seq_col_ptr = *block->get_by_position(_tablet_schema.sequence_col_idx()).column;
+ for (size_t block_pos {0}; block_pos < num_rows; block_pos++) {
+ if (read_index.contains(block_pos)) {
+ new_seq_col_ptr->insert_from(old_seq_col_ptr, read_index[block_pos]);
+ skip_bitmaps[block_pos].remove(seq_col_unique_id);
+ } else {
+ new_seq_col_ptr->insert_from(cur_seq_col_ptr, block_pos);
+ }
+ }
+ block->replace_by_position(_tablet_schema.sequence_col_idx(), std::move(new_seq_col_ptr));
+ return Status::OK();
+}
+
+Status BlockAggregator::aggregate_for_insert_after_delete(
Review Comment:
warning: function 'aggregate_for_insert_after_delete' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
Status BlockAggregator::aggregate_for_insert_after_delete(
^
```
<details>
<summary>Additional context</summary>
**be/src/olap/partial_update_info.cpp:903:** 89 lines including whitespace and comments (threshold 80)
```cpp
Status BlockAggregator::aggregate_for_insert_after_delete(
^
```
</details>
##########
be/src/olap/partial_update_info.cpp:
##########
@@ -658,4 +666,422 @@ Status FlexibleReadPlan::fill_non_primary_key_columns_for_row_store(
return Status::OK();
}
+BlockAggregator::BlockAggregator(segment_v2::VerticalSegmentWriter& vertical_segment_writer)
+ : _writer(vertical_segment_writer), _tablet_schema(*_writer._tablet_schema) {}
+
+void BlockAggregator::merge_one_row(vectorized::MutableBlock& dst_block,
+ vectorized::Block* src_block, int64_t rid,
+ BitmapValue& skip_bitmap) {
+ for (size_t cid {_tablet_schema.num_key_columns()}; cid < _tablet_schema.num_columns(); cid++) {
+ if (cid == _tablet_schema.skip_bitmap_col_idx()) {
+ auto& cur_skip_bitmap =
+ assert_cast<vectorized::ColumnBitmap*>(dst_block.mutable_columns()[cid].get())
+ ->get_data()
+ .back();
+ const auto& new_row_skip_bitmap =
+ assert_cast<vectorized::ColumnBitmap*>(
+ src_block->get_by_position(cid).column->assume_mutable().get())
+ ->get_data()[rid];
+ cur_skip_bitmap &= new_row_skip_bitmap;
+ continue;
+ }
+ if (!skip_bitmap.contains(_tablet_schema.column(cid).unique_id())) {
+ dst_block.mutable_columns()[cid]->pop_back(1);
+ dst_block.mutable_columns()[cid]->insert_from(*src_block->get_by_position(cid).column,
+ rid);
+ }
+ }
+ VLOG_DEBUG << fmt::format("merge a row, after merge, output_block.rows()={}, state: {}",
+ dst_block.rows(), _state.to_string());
+}
+
+void BlockAggregator::append_one_row(vectorized::MutableBlock& dst_block,
+ vectorized::Block* src_block, int64_t rid) {
+ dst_block.add_row(src_block, rid);
+ _state.rows++;
+ VLOG_DEBUG << fmt::format("append a new row, after append, output_block.rows()={}, state: {}",
+ dst_block.rows(), _state.to_string());
+}
+
+void BlockAggregator::remove_last_n_rows(vectorized::MutableBlock& dst_block, int n) {
+ if (n > 0) {
+ for (size_t cid {0}; cid < _tablet_schema.num_columns(); cid++) {
+ DCHECK_GE(dst_block.mutable_columns()[cid]->size(), n);
+ dst_block.mutable_columns()[cid]->pop_back(n);
+ }
+ }
+}
+
+void BlockAggregator::append_or_merge_row(vectorized::MutableBlock& dst_block,
+ vectorized::Block* src_block, int64_t rid,
+ BitmapValue& skip_bitmap, bool have_delete_sign) {
+ if (have_delete_sign) {
+ // remove all the previous batched rows
+ remove_rows(dst_block, _state.rows);
+ _state.rows = 0;
+ _state.has_row_with_delete_sign = true;
+
+ append_one_row(dst_block, src_block, rid);
+ } else {
+ if (_state.should_merge()) {
+ merge_one_row(dst_block, src_block, rid, skip_bitmap);
+ } else {
+ append_one_row(dst_block, src_block, rid);
+ }
+ }
+};
+
+Status BlockAggregator::aggregate_rows(
Review Comment:
warning: function 'aggregate_rows' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
Status BlockAggregator::aggregate_rows(
^
```
<details>
<summary>Additional context</summary>
**be/src/olap/partial_update_info.cpp:733:** 82 lines including whitespace and comments (threshold 80)
```cpp
Status BlockAggregator::aggregate_rows(
^
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2467255562
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix incorrect result when there are insert after delete for same key in one load [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #41701:
URL: https://github.com/apache/doris/pull/41701#discussion_r1805832638
##########
be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:
##########
@@ -558,6 +567,32 @@ Status VerticalSegmentWriter::_append_block_with_partial_content(RowsInBlock& da
return Status::OK();
}
+Status VerticalSegmentWriter::_filter_block(RowsInBlock& data,
+ vectorized::MutableColumnPtr filter_column,
+ int duplicate_rows, std::string col_name) {
+ auto num_cols = data.block->columns();
+ auto* block = const_cast<vectorized::Block*>(data.block);
+ block->insert(
+ {std::move(filter_column), std::make_shared<vectorized::DataTypeUInt8>(), col_name});
+ RETURN_IF_ERROR(vectorized::Block::filter_block(block, num_cols, num_cols));
+ DCHECK_EQ(num_cols, data.block->columns());
+ int merged_rows = data.num_rows - block->rows();
+ VLOG_DEBUG << fmt::format(
+ "filter_block_for_flexible_partial_update[{}] after filter: "
+ "data.block:{}\n",
+ col_name, data.block->dump_data());
+ if (duplicate_rows != merged_rows) {
+ auto msg = fmt::format(
+ "filter_block_for_flexible_partial_update {}: duplicate_rows != merged_rows, "
+ "duplicate_keys={}, merged_rows={}, num_rows={}, mutable_block->rows()={}",
+ col_name, duplicate_rows, merged_rows, data.num_rows, block->rows());
+ DCHECK(false) << msg;
+ return Status::InternalError<false>(msg);
+ }
+ data.num_rows = block->rows();
+ return Status::OK();
+}
+
Status VerticalSegmentWriter::_append_block_with_flexible_partial_content(
Review Comment:
warning: function '_append_block_with_flexible_partial_content' has cognitive complexity of 86 (threshold 50) [readability-function-cognitive-complexity]
```cpp
Status VerticalSegmentWriter::_append_block_with_flexible_partial_content(
^
```
<details>
<summary>Additional context</summary>
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:613:** nesting level increased to 1
```cpp
auto get_skip_bitmaps = [&skip_bitmap_col_idx](const vectorized::Block* block) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:631:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
DBUG_EXECUTE_IF("VerticalSegmentWriter._append_block_with_flexible_partial_content.sleep",
^
```
**be/src/util/debug_points.h:36:** expanded from macro 'DBUG_EXECUTE_IF'
```cpp
if (UNLIKELY(config::enable_debug_points)) { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:631:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
DBUG_EXECUTE_IF("VerticalSegmentWriter._append_block_with_flexible_partial_content.sleep",
^
```
**be/src/util/debug_points.h:38:** expanded from macro 'DBUG_EXECUTE_IF'
```cpp
if (dp) { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:640:** nesting level increased to 1
```cpp
[&full_block, &data,
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:643:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
for (std::size_t cid {0}; cid < _num_sort_key_columns; cid++) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:645:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(_olap_data_convertor->set_source_content_with_specifid_column(
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:645:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
RETURN_IF_ERROR(_olap_data_convertor->set_source_content_with_specifid_column(
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:648:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (!status.ok()) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:656:** nesting level increased to 1
```cpp
auto encode_seq_column = [&data, &schema_has_sequence_col,
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:659:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (schema_has_sequence_col) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:661:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(_olap_data_convertor->set_source_content_with_specifid_column(
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:661:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
RETURN_IF_ERROR(_olap_data_convertor->set_source_content_with_specifid_column(
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:665:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (!status.ok()) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:676:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
RETURN_IF_ERROR(encode_key_columns(key_columns));
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:676:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(encode_key_columns(key_columns));
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:683:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
RETURN_IF_ERROR(encode_seq_column(seq_column));
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:683:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(encode_seq_column(seq_column));
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:690:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
if (schema_has_sequence_col) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:691:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(_merge_rows_for_sequence_column(data, skip_bitmaps, key_columns, seq_column,
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:691:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(_merge_rows_for_sequence_column(data, skip_bitmaps, key_columns, seq_column,
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:694:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (origin_rows != data.num_rows) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:697:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(encode_key_columns(key_columns));
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:697:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
RETURN_IF_ERROR(encode_key_columns(key_columns));
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:698:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(encode_seq_column(seq_column));
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:698:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
RETURN_IF_ERROR(encode_seq_column(seq_column));
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:707:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
RETURN_IF_ERROR(_merge_rows_for_insert_after_delete(
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:707:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(_merge_rows_for_insert_after_delete(
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:709:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
if (data.num_rows != origin_rows) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:712:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(encode_key_columns(key_columns));
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:712:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(encode_key_columns(key_columns));
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:713:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(encode_seq_column(seq_column));
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:713:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(encode_seq_column(seq_column));
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:720:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
for (std::size_t cid {0}; cid < _num_sort_key_columns; cid++) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:723:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(_column_writers[cid]->append(column->get_nullmap(), column->get_data(),
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:723:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(_column_writers[cid]->append(column->get_nullmap(), column->get_data(),
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:731:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
RETURN_IF_ERROR(_generate_flexible_read_plan(
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:731:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(_generate_flexible_read_plan(
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:737:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
if (config::enable_merge_on_write_correctness_check) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:743:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
RETURN_IF_ERROR(read_plan.fill_non_primary_key_columns(
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:743:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(read_plan.fill_non_primary_key_columns(
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:777:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
if (_num_rows_written != data.row_pos ||
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:786:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
RETURN_IF_ERROR(_generate_primary_key_index(_key_coders, key_columns, seq_column, data.num_rows,
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:786:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(_generate_primary_key_index(_key_coders, key_columns, seq_column, data.num_rows,
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
</details>
##########
be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:
##########
@@ -831,98 +884,289 @@
Status VerticalSegmentWriter::_merge_rows_for_sequence_column(
Review Comment:
warning: function '_merge_rows_for_sequence_column' has cognitive complexity of 79 (threshold 50) [readability-function-cognitive-complexity]
```cpp
Status VerticalSegmentWriter::_merge_rows_for_sequence_column(
^
```
<details>
<summary>Additional context</summary>
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:905:** nesting level increased to 1
```cpp
auto append_row = [&](int64_t rid, BitmapValue& skip_bitmap, bool have_delete_sign) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:906:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (have_delete_sign) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:908:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (cur_rows > 0) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:921:** +1, nesting level increased to 2
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:922:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if ((cur_rows == 1 && !have_rows_with_delete_sign) || cur_rows == 2) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:922:** +1
```cpp
if ((cur_rows == 1 && !have_rows_with_delete_sign) || cur_rows == 2) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:922:** +1
```cpp
if ((cur_rows == 1 && !have_rows_with_delete_sign) || cur_rows == 2) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:945:** +1, nesting level increased to 3
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:956:** nesting level increased to 1
```cpp
auto aggregate_rows = [&](std::string key, int64_t start, int64_t end) -> Status {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:958:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (end - start == 1) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:973:** +1
```cpp
DCHECK(st.is<KEY_NOT_FOUND>() || st.ok());
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:976:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (st.ok()) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:977:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
for (int64_t i {start}; i < end; i++) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:983:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
if (!row_has_sequence_col) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:987:** +1, nesting level increased to 4
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:990:** +5, including nesting penalty of 4, nesting level increased to 5
```cpp
if (Slice {seq_val}.compare(Slice {previous_encoded_seq_value}) >= 0) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:997:** +1, nesting level increased to 2
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1001:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (row_has_sequence_col) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1006:** +1, nesting level increased to 3
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1008:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
RETURN_IF_ERROR(_generate_encoded_default_seq_value(
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1008:** +5, including nesting penalty of 4, nesting level increased to 5
```cpp
RETURN_IF_ERROR(_generate_encoded_default_seq_value(
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1015:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
for (int64_t rid {pos}; rid < end; rid++) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1019:** +1
```cpp
(!skip_bitmap.contains(delete_sign_col_unique_id) && delete_signs[rid] != 0);
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1020:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (!row_has_sequence_col) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1022:** +1, nesting level increased to 3
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1025:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
if (Slice {seq_val}.compare(Slice {cur_seq_val}) >= 0) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1028:** +1, nesting level increased to 4
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1039:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
for (size_t block_pos {0}; block_pos < data.num_rows; block_pos++) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1041:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (block_pos > 0 && previous_key == key) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1041:** +1
```cpp
if (block_pos > 0 && previous_key == key) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1043:** +1, nesting level increased to 2
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1044:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (same_key_rows > 0) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1045:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
RETURN_IF_ERROR(aggregate_rows(std::move(previous_key), block_pos - same_key_rows,
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1045:** +5, including nesting penalty of 4, nesting level increased to 5
```cpp
RETURN_IF_ERROR(aggregate_rows(std::move(previous_key), block_pos - same_key_rows,
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1052:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
if (same_key_rows > 0) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1053:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(aggregate_rows(std::move(previous_key), data.num_rows - same_key_rows,
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1053:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(aggregate_rows(std::move(previous_key), data.num_rows - same_key_rows,
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
</details>
##########
be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:
##########
@@ -831,98 +884,289 @@
Status VerticalSegmentWriter::_merge_rows_for_sequence_column(
RowsInBlock& data, std::vector<BitmapValue>* skip_bitmaps,
const std::vector<vectorized::IOlapColumnDataAccessor*>& key_columns,
- vectorized::IOlapColumnDataAccessor* seq_column,
+ vectorized::IOlapColumnDataAccessor* seq_column, const signed char* delete_signs,
const std::vector<RowsetSharedPtr>& specified_rowsets,
std::vector<std::unique_ptr<SegmentCacheHandle>>& segment_caches) {
VLOG_DEBUG << fmt::format(
"VerticalSegmentWriter::_merge_rows_for_sequence_column enter: data.block:{}\n",
data.block->dump_data());
+ // the process logic here is the same as MemTable::_aggregate_for_flexible_partial_update_without_seq_col()
+ // after this function, there will be at most 2 rows for a specified key
+ auto* block = const_cast<vectorized::Block*>(data.block);
auto seq_col_unique_id = _tablet_schema->column(_tablet_schema->sequence_col_idx()).unique_id();
+ auto delete_sign_col_unique_id =
+ _tablet_schema->column(_tablet_schema->delete_sign_idx()).unique_id();
+
+ auto filtered_block = _tablet_schema->create_block();
+ vectorized::MutableBlock output_block =
+ vectorized::MutableBlock::build_mutable_block(&filtered_block);
+ int cur_rows {0};
+ bool have_rows_with_delete_sign {false};
+
+ auto append_row = [&](int64_t rid, BitmapValue& skip_bitmap, bool have_delete_sign) {
+ if (have_delete_sign) {
+ // remove all the preious batched rows
+ if (cur_rows > 0) {
+ for (size_t cid {0}; cid < _tablet_schema->num_columns(); cid++) {
+ DCHECK_GE(output_block.mutable_columns()[cid]->size(), cur_rows);
+ output_block.mutable_columns()[cid]->pop_back(cur_rows);
+ }
+ }
+ output_block.add_row(block, rid);
+ cur_rows = 1;
+ have_rows_with_delete_sign = true;
+ VLOG_DEBUG << fmt::format(
+ "append a row with delete sign, after append, cur_rows={}, "
+ "output_block.rows()={}",
+ cur_rows, output_block.rows());
+ } else {
+ if ((cur_rows == 1 && !have_rows_with_delete_sign) || cur_rows == 2) {
+ for (size_t cid {0}; cid < _tablet_schema->num_columns(); cid++) {
+ if (cid == _tablet_schema->skip_bitmap_col_idx()) {
+ auto& cur_skip_bitmap = assert_cast<vectorized::ColumnBitmap*>(
+ output_block.mutable_columns()[cid].get())
+ ->get_data()
+ .back();
+ const auto& new_row_skip_bitmap =
+ assert_cast<vectorized::ColumnBitmap*>(
+ block->get_by_position(cid).column->assume_mutable().get())
+ ->get_data()[rid];
+ cur_skip_bitmap &= new_row_skip_bitmap;
+ continue;
+ }
+ if (!skip_bitmap.contains(_tablet_schema->column(cid).unique_id())) {
+ output_block.mutable_columns()[cid]->pop_back(1);
+ output_block.mutable_columns()[cid]->insert_from(
+ *block->get_by_position(cid).column, rid);
+ }
+ }
+ VLOG_DEBUG << fmt::format(
+ "merge a row, after merge, cur_rows={}, output_block.rows()={}", cur_rows,
+ output_block.rows());
+ } else {
+ output_block.add_row(block, rid);
+ cur_rows++;
+ VLOG_DEBUG << fmt::format(
+ "append a new row, after append, cur_rows={}, output_block.rows()={}",
+ cur_rows, output_block.rows());
+ }
+ }
+ };
+
+ // aggregate rows with same keys in block range [start, end)
+ auto aggregate_rows = [&](std::string key, int64_t start, int64_t end) -> Status {
+ VLOG_DEBUG << fmt::format("merge rows in range=({}-{})", start, end);
+ if (end - start == 1) {
+ output_block.add_row(block, start);
+ VLOG_DEBUG << fmt::format("append a row directly, rid={}", start);
+ return Status::OK();
+ }
+ cur_rows = 0;
+ have_rows_with_delete_sign = false;
+
+ RowLocation loc;
+ RowsetSharedPtr rowset;
+ std::string previous_encoded_seq_value {};
+ Status st = _tablet->lookup_row_key(key, _tablet_schema.get(), false, specified_rowsets,
+ &loc, _mow_context->max_version, segment_caches,
+ &rowset, true, &previous_encoded_seq_value);
+ int64_t pos = start;
+ DCHECK(st.is<KEY_NOT_FOUND>() || st.ok());
+
+ std::string cur_seq_val;
+ if (st.ok()) {
+ for (int64_t i {start}; i < end; i++) {
+ auto& skip_bitmap = skip_bitmaps->at(i);
+ bool row_has_sequence_col = (!skip_bitmap.contains(seq_col_unique_id));
+ // Find the first row that has larger sequence value than the existing row's
+ // or the first row that doesn't specify the sequence column.
+ // Discard all the rows before and begin to do aggregation from that row
+ if (!row_has_sequence_col) {
+ cur_seq_val = std::move(previous_encoded_seq_value);
+ pos = i;
+ break;
+ } else {
+ std::string seq_val {};
+ _encode_seq_column(seq_column, i, &seq_val);
+ if (Slice {seq_val}.compare(Slice {previous_encoded_seq_value}) >= 0) {
+ cur_seq_val = std::move(seq_val);
+ pos = i;
+ break;
+ }
+ }
+ }
+ } else {
+ pos = start;
+ auto& skip_bitmap = skip_bitmaps->at(pos);
+ bool row_has_sequence_col = (!skip_bitmap.contains(seq_col_unique_id));
+ if (row_has_sequence_col) {
+ std::string seq_val {};
+ // for rows that don't specify seqeunce col, seq_val will be encoded to minial value
+ _encode_seq_column(seq_column, pos, &seq_val);
+ cur_seq_val = std::move(seq_val);
+ } else {
+ cur_seq_val.clear();
+ RETURN_IF_ERROR(_generate_encoded_default_seq_value(
+ *_tablet_schema, *_opts.rowset_ctx->partial_update_info, &cur_seq_val));
+ }
+ }
+ cur_rows = 0;
+ VLOG_DEBUG << fmt::format("start pos={}, output_block.rows()={}", pos, output_block.rows());
+
+ for (int64_t rid {pos}; rid < end; rid++) {
+ auto& skip_bitmap = skip_bitmaps->at(rid);
+ bool row_has_sequence_col = (!skip_bitmap.contains(seq_col_unique_id));
+ bool have_delete_sign =
+ (!skip_bitmap.contains(delete_sign_col_unique_id) && delete_signs[rid] != 0);
+ if (!row_has_sequence_col) {
+ append_row(rid, skip_bitmap, have_delete_sign);
+ } else {
+ std::string seq_val {};
+ _encode_seq_column(seq_column, rid, &seq_val);
+ if (Slice {seq_val}.compare(Slice {cur_seq_val}) >= 0) {
+ append_row(rid, skip_bitmap, have_delete_sign);
+ cur_seq_val = std::move(seq_val);
+ } else {
+ VLOG_DEBUG << fmt::format(
+ "skip rid={} becasue its seq value is lower than the previous", rid);
+ }
+ }
+ }
+ return Status::OK();
+ };
+
+ int same_key_rows {0};
std::string previous_key {};
- bool previous_has_seq_col {false};
- int duplicate_keys {0};
+ for (size_t block_pos {0}; block_pos < data.num_rows; block_pos++) {
+ std::string key = _full_encode_keys(key_columns, block_pos);
+ if (block_pos > 0 && previous_key == key) {
+ same_key_rows++;
+ } else {
+ if (same_key_rows > 0) {
+ RETURN_IF_ERROR(aggregate_rows(std::move(previous_key), block_pos - same_key_rows,
+ block_pos));
+ }
+ same_key_rows = 1;
+ }
+ previous_key = std::move(key);
+ }
+ if (same_key_rows > 0) {
+ RETURN_IF_ERROR(aggregate_rows(std::move(previous_key), data.num_rows - same_key_rows,
+ data.num_rows));
+ }
+ block->swap(output_block.to_block());
+ data.num_rows = block->rows();
+ VLOG_DEBUG << fmt::format(
+ "VerticalSegmentWriter::_merge_rows_for_sequence_column after: data.block:{}\n",
+ data.block->dump_data());
+ return Status::OK();
+}
+
+Status VerticalSegmentWriter::_merge_rows_for_insert_after_delete(
Review Comment:
warning: function '_merge_rows_for_insert_after_delete' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
Status VerticalSegmentWriter::_merge_rows_for_insert_after_delete(
^
```
<details>
<summary>Additional context</summary>
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1065:** 101 lines including whitespace and comments (threshold 80)
```cpp
Status VerticalSegmentWriter::_merge_rows_for_insert_after_delete(
^
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix incorrect result when there are insert after delete for same key in one load [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #41701:
URL: https://github.com/apache/doris/pull/41701#discussion_r1808020918
##########
be/src/olap/partial_update_info.cpp:
##########
@@ -658,4 +669,422 @@
return Status::OK();
}
+BlockAggregator::BlockAggregator(segment_v2::VerticalSegmentWriter& vertical_segment_writer)
+ : _writer(vertical_segment_writer), _tablet_schema(*_writer._tablet_schema) {}
+
+void BlockAggregator::merge_one_row(vectorized::MutableBlock& dst_block,
+ vectorized::Block* src_block, int64_t rid,
+ BitmapValue& skip_bitmap) {
+ for (size_t cid {_tablet_schema.num_key_columns()}; cid < _tablet_schema.num_columns(); cid++) {
+ if (cid == _tablet_schema.skip_bitmap_col_idx()) {
+ auto& cur_skip_bitmap =
+ assert_cast<vectorized::ColumnBitmap*>(dst_block.mutable_columns()[cid].get())
+ ->get_data()
+ .back();
+ const auto& new_row_skip_bitmap =
+ assert_cast<vectorized::ColumnBitmap*>(
+ src_block->get_by_position(cid).column->assume_mutable().get())
+ ->get_data()[rid];
+ cur_skip_bitmap &= new_row_skip_bitmap;
+ continue;
+ }
+ if (!skip_bitmap.contains(_tablet_schema.column(cid).unique_id())) {
+ dst_block.mutable_columns()[cid]->pop_back(1);
+ dst_block.mutable_columns()[cid]->insert_from(*src_block->get_by_position(cid).column,
+ rid);
+ }
+ }
+ VLOG_DEBUG << fmt::format("merge a row, after merge, output_block.rows()={}, state: {}",
+ dst_block.rows(), _state.to_string());
+}
+
+void BlockAggregator::append_one_row(vectorized::MutableBlock& dst_block,
+ vectorized::Block* src_block, int64_t rid) {
+ dst_block.add_row(src_block, rid);
+ _state.rows++;
+ VLOG_DEBUG << fmt::format("append a new row, after append, output_block.rows()={}, state: {}",
+ dst_block.rows(), _state.to_string());
+}
+
+void BlockAggregator::remove_last_n_rows(vectorized::MutableBlock& dst_block, int n) {
+ if (n > 0) {
+ for (size_t cid {0}; cid < _tablet_schema.num_columns(); cid++) {
+ DCHECK_GE(dst_block.mutable_columns()[cid]->size(), n);
+ dst_block.mutable_columns()[cid]->pop_back(n);
+ }
+ }
+}
+
+void BlockAggregator::append_or_merge_row(vectorized::MutableBlock& dst_block,
+ vectorized::Block* src_block, int64_t rid,
+ BitmapValue& skip_bitmap, bool have_delete_sign) {
+ if (have_delete_sign) {
+ // remove all the previous batched rows
+ remove_last_n_rows(dst_block, _state.rows);
+ _state.rows = 0;
+ _state.has_row_with_delete_sign = true;
+
+ append_one_row(dst_block, src_block, rid);
+ } else {
+ if (_state.should_merge()) {
+ merge_one_row(dst_block, src_block, rid, skip_bitmap);
+ } else {
+ append_one_row(dst_block, src_block, rid);
+ }
+ }
+};
+
+Status BlockAggregator::aggregate_rows(
+ vectorized::MutableBlock& output_block, vectorized::Block* block, int64_t start,
+ int64_t end, std::string key, std::vector<BitmapValue>* skip_bitmaps,
+ const signed char* delete_signs, vectorized::IOlapColumnDataAccessor* seq_column,
+ const std::vector<RowsetSharedPtr>& specified_rowsets,
+ std::vector<std::unique_ptr<SegmentCacheHandle>>& segment_caches) {
+ VLOG_DEBUG << fmt::format("merge rows in range=[{}-{})", start, end);
+ if (end - start == 1) {
+ output_block.add_row(block, start);
+ VLOG_DEBUG << fmt::format("append a row directly, rid={}", start);
+ return Status::OK();
+ }
+
+ auto seq_col_unique_id = _tablet_schema.column(_tablet_schema.sequence_col_idx()).unique_id();
+ auto delete_sign_col_unique_id =
+ _tablet_schema.column(_tablet_schema.delete_sign_idx()).unique_id();
+
+ _state.reset();
+
+ RowLocation loc;
+ RowsetSharedPtr rowset;
+ std::string previous_encoded_seq_value {};
+ Status st = _writer._tablet->lookup_row_key(
+ key, &_tablet_schema, false, specified_rowsets, &loc, _writer._mow_context->max_version,
+ segment_caches, &rowset, true, &previous_encoded_seq_value);
+ int64_t pos = start;
+ DCHECK(st.is<ErrorCode::KEY_NOT_FOUND>() || st.ok());
+
+ std::string cur_seq_val;
+ if (st.ok()) {
+ for (int64_t i {start}; i < end; i++) {
+ auto& skip_bitmap = skip_bitmaps->at(i);
+ bool row_has_sequence_col = (!skip_bitmap.contains(seq_col_unique_id));
+ // Find the first row that has larger sequence value than the existing row's
+ // or the first row that doesn't specify the sequence column.
+ // Discard all the rows before and begin to do aggregation from that row
+ if (!row_has_sequence_col) {
+ cur_seq_val = std::move(previous_encoded_seq_value);
+ pos = i;
+ break;
+ } else {
+ std::string seq_val {};
+ _writer._encode_seq_column(seq_column, i, &seq_val);
+ if (Slice {seq_val}.compare(Slice {previous_encoded_seq_value}) >= 0) {
+ cur_seq_val = std::move(seq_val);
+ pos = i;
+ break;
+ }
+ }
+ }
+ } else {
+ pos = start;
+ auto& skip_bitmap = skip_bitmaps->at(pos);
+ bool row_has_sequence_col = (!skip_bitmap.contains(seq_col_unique_id));
+ if (row_has_sequence_col) {
+ std::string seq_val {};
+ // for rows that don't specify seqeunce col, seq_val will be encoded to minial value
+ _writer._encode_seq_column(seq_column, pos, &seq_val);
+ cur_seq_val = std::move(seq_val);
+ } else {
+ cur_seq_val.clear();
+ RETURN_IF_ERROR(_writer._generate_encoded_default_seq_value(
+ _tablet_schema, *_writer._opts.rowset_ctx->partial_update_info, &cur_seq_val));
+ }
+ }
+ VLOG_DEBUG << fmt::format("start pos={}, output_block.rows()={}", pos, output_block.rows());
+
+ for (int64_t rid {pos}; rid < end; rid++) {
+ auto& skip_bitmap = skip_bitmaps->at(rid);
+ bool row_has_sequence_col = (!skip_bitmap.contains(seq_col_unique_id));
+ bool have_delete_sign =
+ (!skip_bitmap.contains(delete_sign_col_unique_id) && delete_signs[rid] != 0);
+ if (!row_has_sequence_col) {
+ append_or_merge_row(output_block, block, rid, skip_bitmap, have_delete_sign);
+ } else {
+ std::string seq_val {};
+ _writer._encode_seq_column(seq_column, rid, &seq_val);
+ if (Slice {seq_val}.compare(Slice {cur_seq_val}) >= 0) {
+ append_or_merge_row(output_block, block, rid, skip_bitmap, have_delete_sign);
+ cur_seq_val = std::move(seq_val);
+ } else {
+ VLOG_DEBUG << fmt::format(
+ "skip rid={} becasue its seq value is lower than the previous", rid);
+ }
+ }
+ }
+ return Status::OK();
+};
+
+Status BlockAggregator::aggregate_for_sequence_column(
+ vectorized::Block* block, size_t num_rows,
+ const std::vector<vectorized::IOlapColumnDataAccessor*>& key_columns,
+ vectorized::IOlapColumnDataAccessor* seq_column,
+ const std::vector<RowsetSharedPtr>& specified_rowsets,
+ std::vector<std::unique_ptr<SegmentCacheHandle>>& segment_caches) {
+ DCHECK_EQ(block->columns(), _tablet_schema.num_columns());
+ VLOG_DEBUG << fmt::format("BlockAggregator::aggregate_for_sequence_column enter: block:{}\n",
+ block->dump_data());
+ // the process logic here is the same as MemTable::_aggregate_for_flexible_partial_update_without_seq_col()
+ // after this function, there will be at most 2 rows for a specified key
+ std::vector<BitmapValue>* skip_bitmaps =
+ &(assert_cast<vectorized::ColumnBitmap*>(
+ block->get_by_position(_tablet_schema.skip_bitmap_col_idx())
+ .column->assume_mutable()
+ .get())
+ ->get_data());
+ const auto* delete_signs = BaseTablet::get_delete_sign_column_data(*block, num_rows);
+
+ auto filtered_block = _tablet_schema.create_block();
+ vectorized::MutableBlock output_block =
+ vectorized::MutableBlock::build_mutable_block(&filtered_block);
+
+ int same_key_rows {0};
+ std::string previous_key {};
+ for (size_t block_pos {0}; block_pos < num_rows; block_pos++) {
+ std::string key = _writer._full_encode_keys(key_columns, block_pos);
+ if (block_pos > 0 && previous_key == key) {
+ same_key_rows++;
+ } else {
+ if (same_key_rows > 0) {
+ RETURN_IF_ERROR(aggregate_rows(output_block, block, block_pos - same_key_rows,
+ block_pos, std::move(previous_key), skip_bitmaps,
+ delete_signs, seq_column, specified_rowsets,
+ segment_caches));
+ }
+ same_key_rows = 1;
+ }
+ previous_key = std::move(key);
+ }
+ if (same_key_rows > 0) {
+ RETURN_IF_ERROR(aggregate_rows(output_block, block, num_rows - same_key_rows, num_rows,
+ std::move(previous_key), skip_bitmaps, delete_signs,
+ seq_column, specified_rowsets, segment_caches));
+ }
+
+ block->swap(output_block.to_block());
+ VLOG_DEBUG << fmt::format(
+ "BlockAggregator::aggregate_for_sequence_column after: data.block:{}\n",
+ block->dump_data());
+ return Status::OK();
+}
+
+Status BlockAggregator::fill_sequence_column(vectorized::Block* block, size_t num_rows,
+ const FixedReadPlan& read_plan,
+ std::vector<BitmapValue>& skip_bitmaps) {
+ DCHECK(_tablet_schema.has_sequence_col());
+ std::vector<uint32_t> cids {static_cast<uint32_t>(_tablet_schema.sequence_col_idx())};
+ auto seq_col_unique_id = _tablet_schema.column(_tablet_schema.sequence_col_idx()).unique_id();
+
+ auto seq_col_block = _tablet_schema.create_block_by_cids(cids);
+ auto tmp_block = _tablet_schema.create_block_by_cids(cids);
+ std::map<uint32_t, uint32_t> read_index;
+ RETURN_IF_ERROR(read_plan.read_columns_by_plan(_tablet_schema, cids, _writer._rsid_to_rowset,
+ seq_col_block, &read_index));
+
+ auto new_seq_col_ptr = tmp_block.get_by_position(0).column->assume_mutable();
+ const auto& old_seq_col_ptr = *seq_col_block.get_by_position(0).column;
+ const auto& cur_seq_col_ptr = *block->get_by_position(_tablet_schema.sequence_col_idx()).column;
+ for (size_t block_pos {0}; block_pos < num_rows; block_pos++) {
+ if (read_index.contains(block_pos)) {
+ new_seq_col_ptr->insert_from(old_seq_col_ptr, read_index[block_pos]);
+ skip_bitmaps[block_pos].remove(seq_col_unique_id);
+ } else {
+ new_seq_col_ptr->insert_from(cur_seq_col_ptr, block_pos);
+ }
+ }
+ block->replace_by_position(_tablet_schema.sequence_col_idx(), std::move(new_seq_col_ptr));
+ return Status::OK();
+}
+
+Status BlockAggregator::aggregate_for_insert_after_delete(
Review Comment:
warning: function 'aggregate_for_insert_after_delete' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
Status BlockAggregator::aggregate_for_insert_after_delete(
^
```
<details>
<summary>Additional context</summary>
**be/src/olap/partial_update_info.cpp:906:** 89 lines including whitespace and comments (threshold 80)
```cpp
Status BlockAggregator::aggregate_for_insert_after_delete(
^
```
</details>
##########
be/src/olap/partial_update_info.cpp:
##########
@@ -658,4 +669,422 @@ Status FlexibleReadPlan::fill_non_primary_key_columns_for_row_store(
return Status::OK();
}
+BlockAggregator::BlockAggregator(segment_v2::VerticalSegmentWriter& vertical_segment_writer)
+ : _writer(vertical_segment_writer), _tablet_schema(*_writer._tablet_schema) {}
+
+void BlockAggregator::merge_one_row(vectorized::MutableBlock& dst_block,
+ vectorized::Block* src_block, int64_t rid,
+ BitmapValue& skip_bitmap) {
+ for (size_t cid {_tablet_schema.num_key_columns()}; cid < _tablet_schema.num_columns(); cid++) {
+ if (cid == _tablet_schema.skip_bitmap_col_idx()) {
+ auto& cur_skip_bitmap =
+ assert_cast<vectorized::ColumnBitmap*>(dst_block.mutable_columns()[cid].get())
+ ->get_data()
+ .back();
+ const auto& new_row_skip_bitmap =
+ assert_cast<vectorized::ColumnBitmap*>(
+ src_block->get_by_position(cid).column->assume_mutable().get())
+ ->get_data()[rid];
+ cur_skip_bitmap &= new_row_skip_bitmap;
+ continue;
+ }
+ if (!skip_bitmap.contains(_tablet_schema.column(cid).unique_id())) {
+ dst_block.mutable_columns()[cid]->pop_back(1);
+ dst_block.mutable_columns()[cid]->insert_from(*src_block->get_by_position(cid).column,
+ rid);
+ }
+ }
+ VLOG_DEBUG << fmt::format("merge a row, after merge, output_block.rows()={}, state: {}",
+ dst_block.rows(), _state.to_string());
+}
+
+void BlockAggregator::append_one_row(vectorized::MutableBlock& dst_block,
+ vectorized::Block* src_block, int64_t rid) {
+ dst_block.add_row(src_block, rid);
+ _state.rows++;
+ VLOG_DEBUG << fmt::format("append a new row, after append, output_block.rows()={}, state: {}",
+ dst_block.rows(), _state.to_string());
+}
+
+void BlockAggregator::remove_last_n_rows(vectorized::MutableBlock& dst_block, int n) {
+ if (n > 0) {
+ for (size_t cid {0}; cid < _tablet_schema.num_columns(); cid++) {
+ DCHECK_GE(dst_block.mutable_columns()[cid]->size(), n);
+ dst_block.mutable_columns()[cid]->pop_back(n);
+ }
+ }
+}
+
+void BlockAggregator::append_or_merge_row(vectorized::MutableBlock& dst_block,
+ vectorized::Block* src_block, int64_t rid,
+ BitmapValue& skip_bitmap, bool have_delete_sign) {
+ if (have_delete_sign) {
+ // remove all the previous batched rows
+ remove_last_n_rows(dst_block, _state.rows);
+ _state.rows = 0;
+ _state.has_row_with_delete_sign = true;
+
+ append_one_row(dst_block, src_block, rid);
+ } else {
+ if (_state.should_merge()) {
+ merge_one_row(dst_block, src_block, rid, skip_bitmap);
+ } else {
+ append_one_row(dst_block, src_block, rid);
+ }
+ }
+};
+
+Status BlockAggregator::aggregate_rows(
Review Comment:
warning: function 'aggregate_rows' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
Status BlockAggregator::aggregate_rows(
^
```
<details>
<summary>Additional context</summary>
**be/src/olap/partial_update_info.cpp:736:** 82 lines including whitespace and comments (threshold 80)
```cpp
Status BlockAggregator::aggregate_rows(
^
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix incorrect result when there are insert after delete for same key in one load [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2425540744
<details>
<summary>TPC-DS: <b>Total hot run time: 192019 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ae261fd99369ad3a45789fe25ed021eca52ebba7, data reload: false
query1 885 393 423 393
query2 6278 2115 2142 2115
query3 8684 194 210 194
query4 34209 23881 23734 23734
query5 3500 483 478 478
query6 294 171 189 171
query7 4194 295 300 295
query8 301 240 237 237
query9 9472 2645 2659 2645
query10 471 277 277 277
query11 17777 15258 15383 15258
query12 159 108 98 98
query13 1574 435 399 399
query14 8882 7065 7429 7065
query15 245 176 175 175
query16 7841 482 457 457
query17 1638 625 593 593
query18 2010 319 314 314
query19 337 155 160 155
query20 121 115 114 114
query21 213 106 109 106
query22 4903 4466 4657 4466
query23 35131 35106 34193 34193
query24 10961 2731 2709 2709
query25 582 411 401 401
query26 900 164 167 164
query27 2312 282 283 282
query28 7581 2433 2409 2409
query29 669 437 424 424
query30 279 154 150 150
query31 1031 813 813 813
query32 99 56 58 56
query33 769 306 300 300
query34 928 511 522 511
query35 884 770 742 742
query36 1115 965 954 954
query37 148 88 88 88
query38 4040 3882 3973 3882
query39 1509 1447 1405 1405
query40 213 99 98 98
query41 49 48 49 48
query42 122 108 98 98
query43 531 498 487 487
query44 1307 820 805 805
query45 202 165 169 165
query46 1137 686 722 686
query47 1984 1840 1872 1840
query48 434 319 335 319
query49 999 441 435 435
query50 818 399 401 399
query51 7024 7167 6927 6927
query52 105 84 86 84
query53 256 179 179 179
query54 1160 443 424 424
query55 80 76 77 76
query56 272 267 257 257
query57 1323 1192 1169 1169
query58 229 254 223 223
query59 3129 3080 2923 2923
query60 298 272 258 258
query61 109 106 99 99
query62 866 655 677 655
query63 225 199 187 187
query64 3905 631 626 626
query65 3266 3157 3196 3157
query66 838 304 309 304
query67 16124 15638 15686 15638
query68 4438 563 563 563
query69 474 278 309 278
query70 1174 1017 1088 1017
query71 355 282 279 279
query72 7017 3906 3983 3906
query73 786 363 364 363
query74 10255 9136 9049 9049
query75 3413 2699 2701 2699
query76 2776 955 1010 955
query77 625 304 292 292
query78 10602 9635 9631 9631
query79 1154 591 603 591
query80 2195 438 458 438
query81 576 240 236 236
query82 736 145 140 140
query83 285 140 135 135
query84 283 71 71 71
query85 1535 311 280 280
query86 380 277 315 277
query87 4499 4330 4409 4330
query88 3369 2203 2167 2167
query89 415 293 285 285
query90 2275 189 190 189
query91 154 101 103 101
query92 65 47 47 47
query93 1069 549 544 544
query94 1158 292 255 255
query95 360 255 251 251
query96 610 282 282 282
query97 3323 3117 3220 3117
query98 233 198 193 193
query99 1605 1270 1296 1270
Total cold run time: 298125 ms
Total hot run time: 192019 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix incorrect result when there are insert after delete for same key in one load [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2413865348
<details>
<summary>TPC-DS: <b>Total hot run time: 192082 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 99e53bd3da17aa9c7e310bf4b8d91beb7a81b396, data reload: false
query1 951 386 393 386
query2 5062 2108 2092 2092
query3 4406 214 220 214
query4 32856 23494 23619 23494
query5 3843 483 468 468
query6 240 175 169 169
query7 3831 299 294 294
query8 286 240 234 234
query9 8374 2781 2743 2743
query10 441 286 274 274
query11 17693 15260 15298 15260
query12 153 99 102 99
query13 1488 445 413 413
query14 10108 7679 7499 7499
query15 292 166 172 166
query16 6715 447 436 436
query17 1528 557 561 557
query18 1838 299 299 299
query19 314 151 145 145
query20 122 107 106 106
query21 213 104 102 102
query22 4634 4392 4266 4266
query23 34799 34155 34033 34033
query24 9899 2799 2835 2799
query25 667 409 403 403
query26 1388 164 159 159
query27 2435 285 282 282
query28 6846 2432 2453 2432
query29 912 435 426 426
query30 327 169 154 154
query31 1056 793 834 793
query32 98 64 63 63
query33 721 297 283 283
query34 940 526 511 511
query35 929 766 803 766
query36 1112 936 950 936
query37 149 98 87 87
query38 4120 3874 3900 3874
query39 1457 1444 1428 1428
query40 286 101 100 100
query41 49 48 44 44
query42 123 103 100 100
query43 541 493 507 493
query44 1210 814 821 814
query45 195 166 164 164
query46 1139 702 702 702
query47 1933 1813 1851 1813
query48 433 320 320 320
query49 1205 417 424 417
query50 816 384 407 384
query51 7030 7028 6983 6983
query52 99 94 88 88
query53 255 180 183 180
query54 917 425 433 425
query55 81 80 75 75
query56 279 268 259 259
query57 1304 1170 1145 1145
query58 237 230 233 230
query59 3290 3127 3108 3108
query60 282 266 261 261
query61 110 106 106 106
query62 882 689 696 689
query63 224 181 187 181
query64 4629 648 599 599
query65 3288 3198 3206 3198
query66 1335 304 301 301
query67 15810 15580 15522 15522
query68 4019 569 558 558
query69 565 275 307 275
query70 1176 1134 1170 1134
query71 362 272 276 272
query72 7090 3977 3936 3936
query73 776 356 359 356
query74 9832 8982 8972 8972
query75 3404 2710 2702 2702
query76 2938 894 969 894
query77 417 288 306 288
query78 10418 9644 9616 9616
query79 3478 609 595 595
query80 1850 447 452 447
query81 590 241 241 241
query82 394 139 145 139
query83 267 137 133 133
query84 266 72 70 70
query85 1146 302 281 281
query86 393 305 300 300
query87 4461 4326 4237 4237
query88 3885 2200 2178 2178
query89 403 321 285 285
query90 1920 185 185 185
query91 145 99 100 99
query92 66 48 48 48
query93 1720 558 559 558
query94 872 294 292 292
query95 337 243 241 241
query96 620 277 276 276
query97 3291 3127 3137 3127
query98 220 198 200 198
query99 1507 1305 1303 1303
Total cold run time: 290053 ms
Total hot run time: 192082 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix incorrect result when there are insert after delete for same key in one load [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2406696909
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website).
See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix incorrect result when there are insert after delete for same key in one load [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #41701:
URL: https://github.com/apache/doris/pull/41701#discussion_r1796916253
##########
be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:
##########
@@ -831,15 +885,18 @@ Status VerticalSegmentWriter::_generate_flexible_read_plan(
Status VerticalSegmentWriter::_merge_rows_for_sequence_column(
Review Comment:
warning: function '_merge_rows_for_sequence_column' has cognitive complexity of 76 (threshold 50) [readability-function-cognitive-complexity]
```cpp
Status VerticalSegmentWriter::_merge_rows_for_sequence_column(
^
```
<details>
<summary>Additional context</summary>
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:905:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
RETURN_IF_ERROR(_generate_encoded_default_seq_value(
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:905:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(_generate_encoded_default_seq_value(
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:916:** nesting level increased to 1
```cpp
auto get_idx = [](bool with_seq_col, bool has_delete_sign) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:917:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (!with_seq_col) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:918:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (has_delete_sign) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:920:** +1, nesting level increased to 3
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:923:** +1, nesting level increased to 2
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:924:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (has_delete_sign) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:926:** +1, nesting level increased to 3
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:932:** nesting level increased to 1
```cpp
auto find_rows_to_filter = [&](const std::string& key) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:935:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (has_row_with_seq_col && has_row_without_seq_col) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:935:** +1
```cpp
if (has_row_with_seq_col && has_row_without_seq_col) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:946:** +1
```cpp
DCHECK(st.is<KEY_NOT_FOUND>() || st.ok());
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:948:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (st.is<KEY_NOT_FOUND>()) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:950:** +1, nesting level increased to 3
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:955:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (batched_rows[2] != -1) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:959:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (batched_rows[3] != -1) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:964:** nesting level increased to 3
```cpp
auto remove_rows_without_seq = [&]() {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:965:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
if (batched_rows[0] != -1) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:970:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
if (batched_rows[1] != -1) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:978:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (batched_rows[2] != -1 && batched_rows[3] != -1) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:979:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
if (previous_seq_slice.compare(Slice {row_with_delete_sign_encoded_seq_value}) <=
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:982:** +1, nesting level increased to 4
```cpp
} else if (previous_seq_slice.compare(
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:988:** +1, nesting level increased to 4
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:996:** +1, nesting level increased to 3
```cpp
} else if (batched_rows[2] != -1) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:997:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
if (previous_seq_slice.compare(Slice {row_with_delete_sign_encoded_seq_value}) <=
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1000:** +1, nesting level increased to 4
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1005:** +1, nesting level increased to 3
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1006:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
if (previous_seq_slice.compare(Slice {row_without_delete_sign_encoded_seq_value}) <=
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1009:** +1, nesting level increased to 4
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1020:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
for (size_t block_pos = data.row_pos; block_pos < data.row_pos + data.num_rows; block_pos++) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1026:** +1
```cpp
(!skip_bitmap.contains(delete_sign_col_unique_id) && delete_signs[block_pos] != 0);
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1029:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (delta_pos > 0 && previous_key == key) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1029:** +1
```cpp
if (delta_pos > 0 && previous_key == key) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1030:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (!has_same_rows) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1036:** +1, nesting level increased to 2
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1037:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (has_same_rows) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1045:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
if (has_same_rows) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1048:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
if (duplicate_keys > 0) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1049:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(_filter_block_for_flexible_partial_update(
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1049:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(_filter_block_for_flexible_partial_update(
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-3044495430
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-3055803868
# Cloud UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/2965d19b21c80c5cb27848aed0ef3365cf87345b_2965d19b21c80c5cb27848aed0ef3365cf87345b_cloud/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/2965d19b21c80c5cb27848aed0ef3365cf87345b_2965d19b21c80c5cb27848aed0ef3365cf87345b_cloud/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 80.60% (1255/1557) |
| Line Coverage | 67.89% (21451/31596) |
| Region Coverage | 67.29% (10614/15774) |
| Branch Coverage | 56.83% (5575/9810) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-3056384569
<details>
<summary>TPC-H: <b>Total hot run time: 33271 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e9db8846be973f1e4fe4bbff061ba3fe85bcf0e9, data reload: false
------ Round 1 ----------------------------------
q1 17567 5317 5011 5011
q2 1933 284 190 190
q3 10357 1358 714 714
q4 10271 1018 514 514
q5 7891 2387 2416 2387
q6 172 159 127 127
q7 906 749 608 608
q8 9303 1379 1126 1126
q9 7019 5194 5113 5113
q10 6896 2387 1939 1939
q11 488 287 277 277
q12 348 345 212 212
q13 17766 3682 3070 3070
q14 221 233 205 205
q15 541 485 475 475
q16 421 421 382 382
q17 571 885 363 363
q18 7791 7235 7155 7155
q19 1232 946 576 576
q20 333 340 215 215
q21 3864 2582 2324 2324
q22 350 315 288 288
Total cold run time: 106241 ms
Total hot run time: 33271 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5066 5037 5054 5037
q2 242 326 225 225
q3 2160 2680 2312 2312
q4 1335 1803 1325 1325
q5 4227 4268 4546 4268
q6 212 169 133 133
q7 2070 1970 1785 1785
q8 2700 2613 2541 2541
q9 7466 7381 7282 7282
q10 3178 3266 2855 2855
q11 597 584 582 582
q12 717 776 610 610
q13 3511 4211 3408 3408
q14 269 287 301 287
q15 515 483 473 473
q16 441 477 430 430
q17 1175 1574 1420 1420
q18 7886 7733 7525 7525
q19 801 847 1031 847
q20 2015 2002 1805 1805
q21 4781 4372 4299 4299
q22 635 582 547 547
Total cold run time: 51999 ms
Total hot run time: 49996 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-3057149592
<details>
<summary>TPC-DS: <b>Total hot run time: 187267 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 49c2fe8f4ea4a7a466b4a04119035ae5a1a9ceaf, data reload: false
query1 987 378 378 378
query2 6501 1726 1747 1726
query3 6738 208 215 208
query4 26958 23350 23141 23141
query5 4331 571 422 422
query6 298 198 197 197
query7 4623 503 284 284
query8 275 215 212 212
query9 8607 2689 2694 2689
query10 451 325 272 272
query11 15269 15046 14862 14862
query12 151 103 114 103
query13 1636 520 397 397
query14 8720 5880 5829 5829
query15 218 207 175 175
query16 7138 645 459 459
query17 941 760 594 594
query18 1991 428 318 318
query19 204 188 162 162
query20 118 118 115 115
query21 209 120 103 103
query22 4307 4710 4540 4540
query23 34863 33900 33530 33530
query24 8480 2393 2419 2393
query25 536 462 381 381
query26 1243 261 142 142
query27 2770 513 354 354
query28 4278 2173 2175 2173
query29 774 537 514 514
query30 283 208 183 183
query31 902 808 710 710
query32 73 62 61 61
query33 551 352 320 320
query34 787 829 509 509
query35 763 814 729 729
query36 935 969 861 861
query37 106 96 71 71
query38 4152 4172 4019 4019
query39 1468 1447 1447 1447
query40 211 119 106 106
query41 57 52 50 50
query42 124 108 110 108
query43 500 496 467 467
query44 1300 834 815 815
query45 179 168 162 162
query46 840 1030 624 624
query47 1792 1848 1759 1759
query48 372 404 309 309
query49 736 481 380 380
query50 626 680 408 408
query51 5375 5567 5464 5464
query52 105 111 101 101
query53 228 248 200 200
query54 596 606 499 499
query55 87 87 83 83
query56 303 307 289 289
query57 1172 1214 1143 1143
query58 258 250 252 250
query59 2537 2709 2687 2687
query60 327 330 302 302
query61 127 139 122 122
query62 798 732 657 657
query63 218 185 187 185
query64 4360 991 642 642
query65 4304 4200 4195 4195
query66 1143 400 346 346
query67 15883 15480 15157 15157
query68 8592 899 531 531
query69 487 322 264 264
query70 1199 1182 1086 1086
query71 480 316 298 298
query72 5676 4785 4751 4751
query73 697 594 357 357
query74 8904 9218 8761 8761
query75 4053 3192 2704 2704
query76 3737 1139 703 703
query77 796 378 293 293
query78 10179 9998 9294 9294
query79 3214 818 588 588
query80 694 499 445 445
query81 466 254 219 219
query82 448 122 93 93
query83 279 255 243 243
query84 299 113 86 86
query85 778 362 318 318
query86 337 318 283 283
query87 4487 4437 4377 4377
query88 3332 2294 2296 2294
query89 449 318 286 286
query90 1946 214 208 208
query91 140 199 111 111
query92 73 63 59 59
query93 2739 958 581 581
query94 670 421 305 305
query95 371 297 287 287
query96 512 563 282 282
query97 2740 2797 2596 2596
query98 234 214 215 214
query99 1458 1374 1257 1257
Total cold run time: 278659 ms
Total hot run time: 187267 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-3051706237
<details>
<summary>TPC-H: <b>Total hot run time: 33330 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit da950276a9235965a5acc2d5ba77e2579c87ae70, data reload: false
------ Round 1 ----------------------------------
q1 17596 5332 5045 5045
q2 1940 314 193 193
q3 10348 1335 728 728
q4 10266 1062 514 514
q5 7991 2373 2360 2360
q6 180 158 128 128
q7 924 750 606 606
q8 9321 1317 1128 1128
q9 7010 5165 5100 5100
q10 6883 2361 1944 1944
q11 474 297 302 297
q12 335 351 206 206
q13 17773 3705 3109 3109
q14 247 228 212 212
q15 556 484 482 482
q16 429 420 389 389
q17 590 865 352 352
q18 7578 7303 7104 7104
q19 1209 963 547 547
q20 337 343 222 222
q21 3754 3163 2381 2381
q22 363 315 283 283
Total cold run time: 106104 ms
Total hot run time: 33330 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5149 5304 5216 5216
q2 240 322 218 218
q3 2179 2693 2256 2256
q4 1371 1825 1395 1395
q5 4208 4374 4550 4374
q6 206 166 145 145
q7 2056 1869 1856 1856
q8 2615 2493 2600 2493
q9 7303 7172 7372 7172
q10 3180 3294 2951 2951
q11 588 510 498 498
q12 657 772 630 630
q13 3561 3993 3424 3424
q14 278 300 293 293
q15 521 484 463 463
q16 455 491 437 437
q17 1190 1597 1400 1400
q18 8224 7685 7764 7685
q19 786 772 884 772
q20 1986 2178 1839 1839
q21 4717 4361 4325 4325
q22 627 593 530 530
Total cold run time: 52097 ms
Total hot run time: 50372 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-3044113896
# Cloud UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/bb350103fa66a6cc5409e2a4761ac6cb9a96d5fd_bb350103fa66a6cc5409e2a4761ac6cb9a96d5fd_cloud/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/bb350103fa66a6cc5409e2a4761ac6cb9a96d5fd_bb350103fa66a6cc5409e2a4761ac6cb9a96d5fd_cloud/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 83.02% (1222/1472) |
| Line Coverage | 67.53% (21174/31355) |
| Region Coverage | 67.27% (10544/15673) |
| Branch Coverage | 56.63% (5549/9798) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2642270270
TeamCity be ut coverage result:
Function Coverage: 42.06% (11025/26215)
Line Coverage: 32.32% (93075/287999)
Region Coverage: 31.48% (47720/151599)
Branch Coverage: 27.49% (24138/87818)
Coverage Report: http://coverage.selectdb-in.cc/coverage/ebcaf82a4f4c0679d1c2fcdb77d2a5355d5a12b6_ebcaf82a4f4c0679d1c2fcdb77d2a5355d5a12b6/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2497314489
<details>
<summary>ClickBench: <b>Total hot run time: 33.61 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 996d2c977562785418ae9b0265c994d07cca4e51, data reload: false
query1 0.04 0.03 0.02
query2 0.08 0.05 0.05
query3 0.23 0.06 0.06
query4 1.64 0.09 0.10
query5 0.43 0.40 0.41
query6 1.15 0.69 0.67
query7 0.02 0.02 0.02
query8 0.05 0.05 0.05
query9 0.54 0.51 0.50
query10 0.55 0.56 0.56
query11 0.17 0.12 0.12
query12 0.16 0.14 0.14
query13 0.61 0.60 0.60
query14 2.75 2.90 2.82
query15 0.94 0.85 0.86
query16 0.37 0.38 0.37
query17 1.08 1.05 1.06
query18 0.19 0.18 0.20
query19 1.85 1.86 2.04
query20 0.02 0.01 0.01
query21 15.35 0.66 0.64
query22 3.82 7.13 2.06
query23 18.27 1.36 1.21
query24 2.07 0.22 0.21
query25 0.15 0.09 0.09
query26 0.28 0.18 0.18
query27 0.08 0.09 0.08
query28 13.33 1.18 1.16
query29 12.65 3.35 3.34
query30 0.25 0.06 0.06
query31 2.87 0.41 0.41
query32 3.23 0.48 0.49
query33 3.09 3.06 3.05
query34 16.74 4.52 4.54
query35 4.63 4.60 4.65
query36 0.67 0.50 0.48
query37 0.19 0.16 0.16
query38 0.15 0.15 0.16
query39 0.06 0.04 0.04
query40 0.15 0.13 0.13
query41 0.09 0.05 0.05
query42 0.06 0.05 0.06
query43 0.06 0.04 0.05
Total cold run time: 111.11 s
Total hot run time: 33.61 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2493477512
<details>
<summary>TPC-DS: <b>Total hot run time: 198566 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 61a7fd52e74446b3f27eccd533514d31d5b431bf, data reload: false
query1 1256 968 970 968
query2 6224 2115 2032 2032
query3 10812 3968 3953 3953
query4 67377 29071 23798 23798
query5 4938 448 448 448
query6 425 190 196 190
query7 5575 307 288 288
query8 302 218 219 218
query9 9069 2753 2739 2739
query10 437 246 242 242
query11 17395 17108 15826 15826
query12 153 108 103 103
query13 1534 442 453 442
query14 10940 7774 7673 7673
query15 219 204 205 204
query16 7562 473 494 473
query17 1050 593 614 593
query18 1916 317 306 306
query19 203 160 161 160
query20 124 117 115 115
query21 216 108 107 107
query22 4696 4512 4620 4512
query23 34838 34417 35292 34417
query24 5535 2598 2556 2556
query25 521 397 402 397
query26 658 153 153 153
query27 1815 283 296 283
query28 4608 2546 2520 2520
query29 675 417 424 417
query30 222 156 155 155
query31 1022 816 842 816
query32 75 56 56 56
query33 437 295 319 295
query34 944 535 519 519
query35 863 742 722 722
query36 1107 954 959 954
query37 126 74 75 74
query38 4467 4458 4372 4372
query39 1543 1491 1473 1473
query40 207 103 96 96
query41 45 41 43 41
query42 111 98 97 97
query43 549 513 493 493
query44 1185 858 841 841
query45 187 170 183 170
query46 1178 696 708 696
query47 2017 1952 1915 1915
query48 429 326 320 320
query49 732 395 396 395
query50 864 413 403 403
query51 7384 7139 7082 7082
query52 100 88 87 87
query53 263 176 178 176
query54 528 407 404 404
query55 75 90 76 76
query56 254 242 232 232
query57 1307 1191 1170 1170
query58 225 221 222 221
query59 3264 3027 3007 3007
query60 268 249 255 249
query61 104 105 104 104
query62 799 669 681 669
query63 218 190 190 190
query64 1424 679 635 635
query65 3360 3214 3189 3189
query66 699 298 309 298
query67 16073 15825 15624 15624
query68 3946 600 587 587
query69 432 254 249 249
query70 1224 1105 1087 1087
query71 360 251 254 251
query72 6266 3912 3982 3912
query73 781 377 372 372
query74 10269 9060 9025 9025
query75 3417 2703 2710 2703
query76 1913 1095 1027 1027
query77 478 275 302 275
query78 10585 9568 9517 9517
query79 1306 593 607 593
query80 848 438 443 438
query81 512 237 235 235
query82 1269 118 121 118
query83 196 151 152 151
query84 285 76 71 71
query85 861 306 308 306
query86 329 297 289 289
query87 4734 4775 4531 4531
query88 3573 2269 2225 2225
query89 428 293 302 293
query90 2030 196 184 184
query91 145 104 106 104
query92 65 50 51 50
query93 1485 564 553 553
query94 768 305 287 287
query95 352 257 245 245
query96 614 286 278 278
query97 2866 2750 2658 2658
query98 215 193 201 193
query99 1665 1316 1310 1310
Total cold run time: 319945 ms
Total hot run time: 198566 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2497271810
<details>
<summary>TPC-H: <b>Total hot run time: 40164 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 996d2c977562785418ae9b0265c994d07cca4e51, data reload: false
------ Round 1 ----------------------------------
q1 17643 7591 7314 7314
q2 2056 185 168 168
q3 10641 1126 1212 1126
q4 10320 741 788 741
q5 7575 2748 2736 2736
q6 238 151 148 148
q7 981 636 620 620
q8 9245 1853 1949 1853
q9 6534 6395 6424 6395
q10 7050 2338 2361 2338
q11 458 259 255 255
q12 430 222 216 216
q13 17766 3027 3076 3027
q14 235 210 212 210
q15 579 540 516 516
q16 676 619 604 604
q17 987 511 515 511
q18 7465 6623 6864 6623
q19 1341 1022 1091 1022
q20 491 182 186 182
q21 4015 3249 3307 3249
q22 389 312 310 310
Total cold run time: 107115 ms
Total hot run time: 40164 ms
----- Round 2, with runtime_filter_mode=off -----
q1 7326 7340 7359 7340
q2 330 234 230 230
q3 2938 2904 2961 2904
q4 2029 1824 1807 1807
q5 5651 5722 5661 5661
q6 226 146 146 146
q7 2230 1838 1829 1829
q8 3426 3601 3567 3567
q9 8774 8905 8945 8905
q10 3621 3581 3584 3581
q11 594 521 534 521
q12 841 609 614 609
q13 10601 3239 3266 3239
q14 298 281 280 280
q15 581 518 525 518
q16 677 652 652 652
q17 1849 1662 1639 1639
q18 8450 7832 7795 7795
q19 1756 1604 1605 1604
q20 2108 1882 1919 1882
q21 5703 5355 5345 5345
q22 622 571 582 571
Total cold run time: 70631 ms
Total hot run time: 60625 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2609784606
<details>
<summary>ClickBench: <b>Total hot run time: 31.06 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 01f1538e375d5d84cf26481834e2500342d61174, data reload: false
query1 0.04 0.03 0.03
query2 0.09 0.04 0.04
query3 0.23 0.05 0.05
query4 1.64 0.07 0.08
query5 0.42 0.39 0.40
query6 1.16 0.66 0.67
query7 0.02 0.01 0.01
query8 0.05 0.05 0.05
query9 0.55 0.50 0.49
query10 0.55 0.57 0.56
query11 0.16 0.12 0.11
query12 0.16 0.12 0.13
query13 0.60 0.60 0.60
query14 2.74 2.84 2.83
query15 0.91 0.83 0.82
query16 0.38 0.39 0.39
query17 0.97 1.02 1.00
query18 0.19 0.19 0.18
query19 1.86 1.92 2.06
query20 0.01 0.01 0.01
query21 15.37 0.98 0.66
query22 0.77 0.79 0.72
query23 14.93 1.50 0.70
query24 2.14 0.36 0.22
query25 0.14 0.09 0.08
query26 0.28 0.19 0.18
query27 0.08 0.09 0.09
query28 13.47 1.29 0.56
query29 12.60 4.12 3.36
query30 0.25 0.08 0.06
query31 2.84 0.61 0.41
query32 3.22 0.58 0.48
query33 3.00 3.06 3.02
query34 16.67 5.12 4.54
query35 4.58 4.56 4.62
query36 0.81 0.47 0.50
query37 0.20 0.16 0.16
query38 0.16 0.15 0.15
query39 0.05 0.04 0.04
query40 0.17 0.14 0.13
query41 0.10 0.06 0.05
query42 0.06 0.05 0.05
query43 0.05 0.05 0.04
Total cold run time: 104.67 s
Total hot run time: 31.06 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix incorrect result when there are insert after delete for same key in one load [doris]
Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2443297230
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix incorrect result when there are insert after delete for same key in one load [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #41701:
URL: https://github.com/apache/doris/pull/41701#discussion_r1825400400
##########
be/src/olap/partial_update_info.cpp:
##########
@@ -658,4 +669,431 @@ Status FlexibleReadPlan::fill_non_primary_key_columns_for_row_store(
return Status::OK();
}
+BlockAggregator::BlockAggregator(segment_v2::VerticalSegmentWriter& vertical_segment_writer)
+ : _writer(vertical_segment_writer), _tablet_schema(*_writer._tablet_schema) {}
+
+void BlockAggregator::merge_one_row(vectorized::MutableBlock& dst_block,
+ vectorized::Block* src_block, int64_t rid,
+ BitmapValue& skip_bitmap) {
+ for (size_t cid {_tablet_schema.num_key_columns()}; cid < _tablet_schema.num_columns(); cid++) {
+ if (cid == _tablet_schema.skip_bitmap_col_idx()) {
+ auto& cur_skip_bitmap =
+ assert_cast<vectorized::ColumnBitmap*>(dst_block.mutable_columns()[cid].get())
+ ->get_data()
+ .back();
+ const auto& new_row_skip_bitmap =
+ assert_cast<vectorized::ColumnBitmap*>(
+ src_block->get_by_position(cid).column->assume_mutable().get())
+ ->get_data()[rid];
+ cur_skip_bitmap &= new_row_skip_bitmap;
+ continue;
+ }
+ if (!skip_bitmap.contains(_tablet_schema.column(cid).unique_id())) {
+ dst_block.mutable_columns()[cid]->pop_back(1);
+ dst_block.mutable_columns()[cid]->insert_from(*src_block->get_by_position(cid).column,
+ rid);
+ }
+ }
+ VLOG_DEBUG << fmt::format("merge a row, after merge, output_block.rows()={}, state: {}",
+ dst_block.rows(), _state.to_string());
+}
+
+void BlockAggregator::append_one_row(vectorized::MutableBlock& dst_block,
+ vectorized::Block* src_block, int64_t rid) {
+ dst_block.add_row(src_block, rid);
+ _state.rows++;
+ VLOG_DEBUG << fmt::format("append a new row, after append, output_block.rows()={}, state: {}",
+ dst_block.rows(), _state.to_string());
+}
+
+void BlockAggregator::remove_last_n_rows(vectorized::MutableBlock& dst_block, int n) {
+ if (n > 0) {
+ for (size_t cid {0}; cid < _tablet_schema.num_columns(); cid++) {
+ DCHECK_GE(dst_block.mutable_columns()[cid]->size(), n);
+ dst_block.mutable_columns()[cid]->pop_back(n);
+ }
+ }
+}
+
+void BlockAggregator::append_or_merge_row(vectorized::MutableBlock& dst_block,
+ vectorized::Block* src_block, int64_t rid,
+ BitmapValue& skip_bitmap, bool have_delete_sign) {
+ if (have_delete_sign) {
+ // remove all the previous batched rows
+ remove_last_n_rows(dst_block, _state.rows);
+ _state.rows = 0;
+ _state.has_row_with_delete_sign = true;
+
+ append_one_row(dst_block, src_block, rid);
+ } else {
+ if (_state.should_merge()) {
+ merge_one_row(dst_block, src_block, rid, skip_bitmap);
+ } else {
+ append_one_row(dst_block, src_block, rid);
+ }
+ }
+};
+
+Status BlockAggregator::aggregate_rows(
Review Comment:
warning: function 'aggregate_rows' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
Status BlockAggregator::aggregate_rows(
^
```
<details>
<summary>Additional context</summary>
**be/src/olap/partial_update_info.cpp:736:** 84 lines including whitespace and comments (threshold 80)
```cpp
Status BlockAggregator::aggregate_rows(
^
```
</details>
##########
be/src/olap/partial_update_info.cpp:
##########
@@ -658,4 +669,431 @@
return Status::OK();
}
+BlockAggregator::BlockAggregator(segment_v2::VerticalSegmentWriter& vertical_segment_writer)
+ : _writer(vertical_segment_writer), _tablet_schema(*_writer._tablet_schema) {}
+
+void BlockAggregator::merge_one_row(vectorized::MutableBlock& dst_block,
+ vectorized::Block* src_block, int64_t rid,
+ BitmapValue& skip_bitmap) {
+ for (size_t cid {_tablet_schema.num_key_columns()}; cid < _tablet_schema.num_columns(); cid++) {
+ if (cid == _tablet_schema.skip_bitmap_col_idx()) {
+ auto& cur_skip_bitmap =
+ assert_cast<vectorized::ColumnBitmap*>(dst_block.mutable_columns()[cid].get())
+ ->get_data()
+ .back();
+ const auto& new_row_skip_bitmap =
+ assert_cast<vectorized::ColumnBitmap*>(
+ src_block->get_by_position(cid).column->assume_mutable().get())
+ ->get_data()[rid];
+ cur_skip_bitmap &= new_row_skip_bitmap;
+ continue;
+ }
+ if (!skip_bitmap.contains(_tablet_schema.column(cid).unique_id())) {
+ dst_block.mutable_columns()[cid]->pop_back(1);
+ dst_block.mutable_columns()[cid]->insert_from(*src_block->get_by_position(cid).column,
+ rid);
+ }
+ }
+ VLOG_DEBUG << fmt::format("merge a row, after merge, output_block.rows()={}, state: {}",
+ dst_block.rows(), _state.to_string());
+}
+
+void BlockAggregator::append_one_row(vectorized::MutableBlock& dst_block,
+ vectorized::Block* src_block, int64_t rid) {
+ dst_block.add_row(src_block, rid);
+ _state.rows++;
+ VLOG_DEBUG << fmt::format("append a new row, after append, output_block.rows()={}, state: {}",
+ dst_block.rows(), _state.to_string());
+}
+
+void BlockAggregator::remove_last_n_rows(vectorized::MutableBlock& dst_block, int n) {
+ if (n > 0) {
+ for (size_t cid {0}; cid < _tablet_schema.num_columns(); cid++) {
+ DCHECK_GE(dst_block.mutable_columns()[cid]->size(), n);
+ dst_block.mutable_columns()[cid]->pop_back(n);
+ }
+ }
+}
+
+void BlockAggregator::append_or_merge_row(vectorized::MutableBlock& dst_block,
+ vectorized::Block* src_block, int64_t rid,
+ BitmapValue& skip_bitmap, bool have_delete_sign) {
+ if (have_delete_sign) {
+ // remove all the previous batched rows
+ remove_last_n_rows(dst_block, _state.rows);
+ _state.rows = 0;
+ _state.has_row_with_delete_sign = true;
+
+ append_one_row(dst_block, src_block, rid);
+ } else {
+ if (_state.should_merge()) {
+ merge_one_row(dst_block, src_block, rid, skip_bitmap);
+ } else {
+ append_one_row(dst_block, src_block, rid);
+ }
+ }
+};
+
+Status BlockAggregator::aggregate_rows(
+ vectorized::MutableBlock& output_block, vectorized::Block* block, int64_t start,
+ int64_t end, std::string key, std::vector<BitmapValue>* skip_bitmaps,
+ const signed char* delete_signs, vectorized::IOlapColumnDataAccessor* seq_column,
+ const std::vector<RowsetSharedPtr>& specified_rowsets,
+ std::vector<std::unique_ptr<SegmentCacheHandle>>& segment_caches) {
+ VLOG_DEBUG << fmt::format("merge rows in range=[{}-{})", start, end);
+ if (end - start == 1) {
+ output_block.add_row(block, start);
+ VLOG_DEBUG << fmt::format("append a row directly, rid={}", start);
+ return Status::OK();
+ }
+
+ auto seq_col_unique_id = _tablet_schema.column(_tablet_schema.sequence_col_idx()).unique_id();
+ auto delete_sign_col_unique_id =
+ _tablet_schema.column(_tablet_schema.delete_sign_idx()).unique_id();
+
+ _state.reset();
+
+ RowLocation loc;
+ RowsetSharedPtr rowset;
+ std::string previous_encoded_seq_value {};
+ Status st = _writer._tablet->lookup_row_key(
+ key, &_tablet_schema, false, specified_rowsets, &loc, _writer._mow_context->max_version,
+ segment_caches, &rowset, true, &previous_encoded_seq_value);
+ int64_t pos = start;
+ bool is_expected_st = (st.is<ErrorCode::KEY_NOT_FOUND>() || st.ok());
+ DCHECK(is_expected_st || st.is<ErrorCode::MEM_LIMIT_EXCEEDED>())
+ << "[BlockAggregator::aggregate_rows] unexpected error status while lookup_row_key:"
+ << st;
+ if (!is_expected_st) {
+ return st;
+ }
+
+ std::string cur_seq_val;
+ if (st.ok()) {
+ for (pos = start; pos < end; pos++) {
+ auto& skip_bitmap = skip_bitmaps->at(pos);
+ bool row_has_sequence_col = (!skip_bitmap.contains(seq_col_unique_id));
+ // Discard all the rows whose seq value is smaller than previous_encoded_seq_value.
+ if (row_has_sequence_col) {
+ std::string seq_val {};
+ _writer._encode_seq_column(seq_column, pos, &seq_val);
+ if (Slice {seq_val}.compare(Slice {previous_encoded_seq_value}) < 0) {
+ continue;
+ }
+ cur_seq_val = std::move(seq_val);
+ break;
+ }
+ cur_seq_val = std::move(previous_encoded_seq_value);
+ break;
+ }
+ } else {
+ pos = start;
+ auto& skip_bitmap = skip_bitmaps->at(pos);
+ bool row_has_sequence_col = (!skip_bitmap.contains(seq_col_unique_id));
+ if (row_has_sequence_col) {
+ std::string seq_val {};
+ // for rows that don't specify seqeunce col, seq_val will be encoded to minial value
+ _writer._encode_seq_column(seq_column, pos, &seq_val);
+ cur_seq_val = std::move(seq_val);
+ } else {
+ cur_seq_val.clear();
+ RETURN_IF_ERROR(_writer._generate_encoded_default_seq_value(
+ _tablet_schema, *_writer._opts.rowset_ctx->partial_update_info, &cur_seq_val));
+ }
+ }
+ VLOG_DEBUG << fmt::format("start pos={}, output_block.rows()={}", pos, output_block.rows());
+
+ for (int64_t rid {pos}; rid < end; rid++) {
+ auto& skip_bitmap = skip_bitmaps->at(rid);
+ bool row_has_sequence_col = (!skip_bitmap.contains(seq_col_unique_id));
+ bool have_delete_sign =
+ (!skip_bitmap.contains(delete_sign_col_unique_id) && delete_signs[rid] != 0);
+ if (!row_has_sequence_col) {
+ append_or_merge_row(output_block, block, rid, skip_bitmap, have_delete_sign);
+ } else {
+ std::string seq_val {};
+ _writer._encode_seq_column(seq_column, rid, &seq_val);
+ if (Slice {seq_val}.compare(Slice {cur_seq_val}) >= 0) {
+ append_or_merge_row(output_block, block, rid, skip_bitmap, have_delete_sign);
+ cur_seq_val = std::move(seq_val);
+ } else {
+ VLOG_DEBUG << fmt::format(
+ "skip rid={} becasue its seq value is lower than the previous", rid);
+ }
+ }
+ }
+ return Status::OK();
+};
+
+Status BlockAggregator::aggregate_for_sequence_column(
+ vectorized::Block* block, size_t num_rows,
+ const std::vector<vectorized::IOlapColumnDataAccessor*>& key_columns,
+ vectorized::IOlapColumnDataAccessor* seq_column,
+ const std::vector<RowsetSharedPtr>& specified_rowsets,
+ std::vector<std::unique_ptr<SegmentCacheHandle>>& segment_caches) {
+ DCHECK_EQ(block->columns(), _tablet_schema.num_columns());
+ VLOG_DEBUG << fmt::format("BlockAggregator::aggregate_for_sequence_column enter: block:{}\n",
+ block->dump_data());
+ // the process logic here is the same as MemTable::_aggregate_for_flexible_partial_update_without_seq_col()
+ // after this function, there will be at most 2 rows for a specified key
+ std::vector<BitmapValue>* skip_bitmaps =
+ &(assert_cast<vectorized::ColumnBitmap*>(
+ block->get_by_position(_tablet_schema.skip_bitmap_col_idx())
+ .column->assume_mutable()
+ .get())
+ ->get_data());
+ const auto* delete_signs = BaseTablet::get_delete_sign_column_data(*block, num_rows);
+
+ auto filtered_block = _tablet_schema.create_block();
+ vectorized::MutableBlock output_block =
+ vectorized::MutableBlock::build_mutable_block(&filtered_block);
+
+ int same_key_rows {0};
+ std::string previous_key {};
+ for (size_t block_pos {0}; block_pos < num_rows; block_pos++) {
+ std::string key = _writer._full_encode_keys(key_columns, block_pos);
+ if (block_pos > 0 && previous_key == key) {
+ same_key_rows++;
+ } else {
+ if (same_key_rows > 0) {
+ RETURN_IF_ERROR(aggregate_rows(output_block, block, block_pos - same_key_rows,
+ block_pos, std::move(previous_key), skip_bitmaps,
+ delete_signs, seq_column, specified_rowsets,
+ segment_caches));
+ }
+ same_key_rows = 1;
+ }
+ previous_key = std::move(key);
+ }
+ if (same_key_rows > 0) {
+ RETURN_IF_ERROR(aggregate_rows(output_block, block, num_rows - same_key_rows, num_rows,
+ std::move(previous_key), skip_bitmaps, delete_signs,
+ seq_column, specified_rowsets, segment_caches));
+ }
+
+ block->swap(output_block.to_block());
+ VLOG_DEBUG << fmt::format(
+ "BlockAggregator::aggregate_for_sequence_column after: data.block:{}\n",
+ block->dump_data());
+ return Status::OK();
+}
+
+Status BlockAggregator::fill_sequence_column(vectorized::Block* block, size_t num_rows,
+ const FixedReadPlan& read_plan,
+ std::vector<BitmapValue>& skip_bitmaps) {
+ DCHECK(_tablet_schema.has_sequence_col());
+ std::vector<uint32_t> cids {static_cast<uint32_t>(_tablet_schema.sequence_col_idx())};
+ auto seq_col_unique_id = _tablet_schema.column(_tablet_schema.sequence_col_idx()).unique_id();
+
+ auto seq_col_block = _tablet_schema.create_block_by_cids(cids);
+ auto tmp_block = _tablet_schema.create_block_by_cids(cids);
+ std::map<uint32_t, uint32_t> read_index;
+ RETURN_IF_ERROR(read_plan.read_columns_by_plan(_tablet_schema, cids, _writer._rsid_to_rowset,
+ seq_col_block, &read_index));
+
+ auto new_seq_col_ptr = tmp_block.get_by_position(0).column->assume_mutable();
+ const auto& old_seq_col_ptr = *seq_col_block.get_by_position(0).column;
+ const auto& cur_seq_col_ptr = *block->get_by_position(_tablet_schema.sequence_col_idx()).column;
+ for (size_t block_pos {0}; block_pos < num_rows; block_pos++) {
+ if (read_index.contains(block_pos)) {
+ new_seq_col_ptr->insert_from(old_seq_col_ptr, read_index[block_pos]);
+ skip_bitmaps[block_pos].remove(seq_col_unique_id);
+ } else {
+ new_seq_col_ptr->insert_from(cur_seq_col_ptr, block_pos);
+ }
+ }
+ block->replace_by_position(_tablet_schema.sequence_col_idx(), std::move(new_seq_col_ptr));
+ return Status::OK();
+}
+
+Status BlockAggregator::aggregate_for_insert_after_delete(
Review Comment:
warning: function 'aggregate_for_insert_after_delete' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
Status BlockAggregator::aggregate_for_insert_after_delete(
^
```
<details>
<summary>Additional context</summary>
**be/src/olap/partial_update_info.cpp:908:** 96 lines including whitespace and comments (threshold 80)
```cpp
Status BlockAggregator::aggregate_for_insert_after_delete(
^
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2467295779
TeamCity be ut coverage result:
Function Coverage: 37.88% (9867/26047)
Line Coverage: 29.05% (82244/283147)
Region Coverage: 28.20% (42342/150156)
Branch Coverage: 24.74% (21437/86638)
Coverage Report: http://coverage.selectdb-in.cc/coverage/a35d1360e38a6d95dbf89a478f03033af139bd79_a35d1360e38a6d95dbf89a478f03033af139bd79/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2467148146
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2595473865
<details>
<summary>ClickBench: <b>Total hot run time: 30.02 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 21ec0541fbd8c9eab4a761c02f69ddb72c4856a6, data reload: false
query1 0.04 0.04 0.03
query2 0.07 0.03 0.03
query3 0.24 0.08 0.07
query4 1.61 0.11 0.11
query5 0.43 0.41 0.42
query6 1.17 0.65 0.64
query7 0.02 0.02 0.01
query8 0.04 0.03 0.03
query9 0.60 0.49 0.53
query10 0.53 0.56 0.56
query11 0.14 0.10 0.10
query12 0.14 0.11 0.11
query13 0.62 0.59 0.59
query14 2.69 2.71 2.79
query15 0.89 0.83 0.83
query16 0.39 0.38 0.38
query17 1.07 1.04 1.05
query18 0.22 0.20 0.21
query19 1.84 1.76 2.00
query20 0.02 0.01 0.01
query21 15.42 0.92 0.58
query22 0.75 0.67 0.62
query23 15.44 1.40 0.50
query24 2.95 0.58 1.06
query25 0.20 0.17 0.07
query26 0.42 0.16 0.15
query27 0.08 0.05 0.05
query28 13.13 1.08 0.44
query29 12.95 3.97 3.29
query30 0.25 0.10 0.06
query31 2.85 0.60 0.39
query32 3.23 0.55 0.47
query33 3.01 2.98 3.08
query34 16.84 5.10 4.45
query35 4.52 4.49 4.52
query36 0.63 0.48 0.48
query37 0.09 0.06 0.06
query38 0.05 0.04 0.04
query39 0.03 0.02 0.03
query40 0.18 0.14 0.12
query41 0.08 0.03 0.02
query42 0.04 0.02 0.03
query43 0.03 0.03 0.03
Total cold run time: 105.94 s
Total hot run time: 30.02 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2597302816
run external
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2556222720
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2561933101
<details>
<summary>TPC-H: <b>Total hot run time: 32706 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9cc0eab95240250a54bda6aa63dfd1cf1932c83e, data reload: false
------ Round 1 ----------------------------------
q1 17584 6144 6074 6074
q2 2048 306 179 179
q3 10409 1311 728 728
q4 10217 867 427 427
q5 7538 2250 2036 2036
q6 211 187 148 148
q7 903 767 621 621
q8 9231 1414 1189 1189
q9 5219 4943 4844 4844
q10 6750 2297 1847 1847
q11 470 275 268 268
q12 358 358 217 217
q13 17757 3639 2997 2997
q14 237 236 220 220
q15 563 503 512 503
q16 629 631 595 595
q17 581 864 325 325
q18 7100 6516 6475 6475
q19 1584 989 547 547
q20 305 325 185 185
q21 2840 2168 1972 1972
q22 365 336 309 309
Total cold run time: 102899 ms
Total hot run time: 32706 ms
----- Round 2, with runtime_filter_mode=off -----
q1 6273 6226 6217 6217
q2 239 330 236 236
q3 2229 2635 2332 2332
q4 1438 1879 1331 1331
q5 4374 4779 4664 4664
q6 182 173 139 139
q7 1977 1851 1749 1749
q8 2518 2692 2647 2647
q9 7009 6839 6902 6839
q10 2939 3217 2704 2704
q11 588 520 494 494
q12 647 696 566 566
q13 3206 3668 2993 2993
q14 272 299 273 273
q15 557 522 502 502
q16 655 664 647 647
q17 1185 1706 1244 1244
q18 7570 7133 6946 6946
q19 811 1109 1085 1085
q20 1938 1982 1848 1848
q21 5492 5092 4792 4792
q22 612 623 603 603
Total cold run time: 52711 ms
Total hot run time: 50851 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-3079198795
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-3055974876
# Cloud UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/e9db8846be973f1e4fe4bbff061ba3fe85bcf0e9_e9db8846be973f1e4fe4bbff061ba3fe85bcf0e9_cloud/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/e9db8846be973f1e4fe4bbff061ba3fe85bcf0e9_e9db8846be973f1e4fe4bbff061ba3fe85bcf0e9_cloud/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 80.60% (1255/1557) |
| Line Coverage | 67.76% (21411/31596) |
| Region Coverage | 67.24% (10607/15774) |
| Branch Coverage | 56.70% (5562/9810) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix incorrect result when there are insert after delete for same key in one load [doris]
Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2443823512
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix incorrect result when there are insert after delete for same key in one load [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2443922048
<details>
<summary>TPC-H: <b>Total hot run time: 41436 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d978d67bc61473d4e3aeadc3ec15eff34e8f7593, data reload: false
------ Round 1 ----------------------------------
q1 17587 7542 7345 7345
q2 2045 167 183 167
q3 10791 1055 1181 1055
q4 10523 844 814 814
q5 7772 3088 3040 3040
q6 235 147 144 144
q7 1014 611 591 591
q8 9361 1937 2019 1937
q9 6661 6496 6534 6496
q10 7077 2458 2437 2437
q11 463 250 250 250
q12 415 224 218 218
q13 17788 3002 3017 3002
q14 253 216 221 216
q15 582 502 527 502
q16 667 591 585 585
q17 983 514 607 514
q18 7376 6670 6653 6653
q19 1335 1041 1009 1009
q20 474 190 185 185
q21 4051 3249 3259 3249
q22 1125 1046 1027 1027
Total cold run time: 108578 ms
Total hot run time: 41436 ms
----- Round 2, with runtime_filter_mode=off -----
q1 7312 7300 7286 7286
q2 326 227 229 227
q3 3036 2926 3004 2926
q4 2155 1880 1837 1837
q5 5718 5770 5817 5770
q6 223 138 139 138
q7 2273 1792 1792 1792
q8 3376 3534 3437 3437
q9 8930 8935 8880 8880
q10 3573 3578 3526 3526
q11 592 489 485 485
q12 861 664 603 603
q13 10622 3159 3176 3159
q14 298 272 265 265
q15 572 544 525 525
q16 682 648 653 648
q17 1872 1617 1598 1598
q18 8458 7634 7694 7634
q19 1725 1567 1604 1567
q20 2121 1883 1875 1875
q21 5454 5483 5390 5390
q22 1179 1101 1060 1060
Total cold run time: 71358 ms
Total hot run time: 60628 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix incorrect result when there are insert after delete for same key in one load [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2451451345
<details>
<summary>TPC-H: <b>Total hot run time: 41622 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ffefc7377f05b2b19319afb73bc7375cd0370491, data reload: false
------ Round 1 ----------------------------------
q1 17945 8411 7363 7363
q2 2055 181 161 161
q3 10679 1088 1254 1088
q4 10554 848 893 848
q5 7755 3143 3096 3096
q6 238 153 156 153
q7 1021 622 626 622
q8 9367 2041 2078 2041
q9 6683 6466 6451 6451
q10 7083 2429 2425 2425
q11 463 261 259 259
q12 417 216 221 216
q13 17785 3026 2989 2989
q14 238 205 208 205
q15 571 510 512 510
q16 644 579 564 564
q17 982 500 543 500
q18 7450 6810 6747 6747
q19 1322 1010 1003 1003
q20 482 180 181 180
q21 3995 3360 3183 3183
q22 1128 1038 1018 1018
Total cold run time: 108857 ms
Total hot run time: 41622 ms
----- Round 2, with runtime_filter_mode=off -----
q1 7398 7336 7239 7239
q2 324 237 235 235
q3 3119 2954 3017 2954
q4 2104 1854 1740 1740
q5 5799 5794 5841 5794
q6 237 147 142 142
q7 2280 1778 1867 1778
q8 3373 3438 3536 3438
q9 8971 9021 8933 8933
q10 3613 3573 3593 3573
q11 621 506 500 500
q12 828 603 611 603
q13 9488 3192 3228 3192
q14 306 294 292 292
q15 573 527 531 527
q16 706 661 658 658
q17 1850 1653 1641 1641
q18 8414 7831 7671 7671
q19 1742 1599 1678 1599
q20 2154 1847 1897 1847
q21 5694 5428 5465 5428
q22 1168 1042 1065 1042
Total cold run time: 70762 ms
Total hot run time: 60826 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix incorrect result when there are insert after delete for same key in one load [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2408444770
TeamCity be ut coverage result:
Function Coverage: 37.39% (9666/25854)
Line Coverage: 28.68% (80375/280236)
Region Coverage: 28.06% (41451/147730)
Branch Coverage: 24.68% (21113/85560)
Coverage Report: http://coverage.selectdb-in.cc/coverage/f177418f2be6c8d0161d7df4ecbf5451fd3791c0_f177418f2be6c8d0161d7df4ecbf5451fd3791c0/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2464135956
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix incorrect result when there are insert after delete for same key in one load [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #41701:
URL: https://github.com/apache/doris/pull/41701#discussion_r1820147414
##########
be/src/olap/partial_update_info.cpp:
##########
@@ -658,4 +664,433 @@
return Status::OK();
}
+BlockAggregator::BlockAggregator(segment_v2::VerticalSegmentWriter& vertical_segment_writer)
+ : _writer(vertical_segment_writer), _tablet_schema(*_writer._tablet_schema) {}
+
+void BlockAggregator::merge_one_row(vectorized::MutableBlock& dst_block,
+ vectorized::Block* src_block, int64_t rid,
+ BitmapValue& skip_bitmap) {
+ for (size_t cid {_tablet_schema.num_key_columns()}; cid < _tablet_schema.num_columns(); cid++) {
+ if (cid == _tablet_schema.skip_bitmap_col_idx()) {
+ auto& cur_skip_bitmap =
+ assert_cast<vectorized::ColumnBitmap*>(dst_block.mutable_columns()[cid].get())
+ ->get_data()
+ .back();
+ const auto& new_row_skip_bitmap =
+ assert_cast<vectorized::ColumnBitmap*>(
+ src_block->get_by_position(cid).column->assume_mutable().get())
+ ->get_data()[rid];
+ cur_skip_bitmap &= new_row_skip_bitmap;
+ continue;
+ }
+ if (!skip_bitmap.contains(_tablet_schema.column(cid).unique_id())) {
+ dst_block.mutable_columns()[cid]->pop_back(1);
+ dst_block.mutable_columns()[cid]->insert_from(*src_block->get_by_position(cid).column,
+ rid);
+ }
+ }
+ VLOG_DEBUG << fmt::format("merge a row, after merge, output_block.rows()={}, state: {}",
+ dst_block.rows(), _state.to_string());
+}
+
+void BlockAggregator::append_one_row(vectorized::MutableBlock& dst_block,
+ vectorized::Block* src_block, int64_t rid) {
+ dst_block.add_row(src_block, rid);
+ _state.rows++;
+ VLOG_DEBUG << fmt::format("append a new row, after append, output_block.rows()={}, state: {}",
+ dst_block.rows(), _state.to_string());
+}
+
+void BlockAggregator::remove_last_n_rows(vectorized::MutableBlock& dst_block, int n) {
+ if (n > 0) {
+ for (size_t cid {0}; cid < _tablet_schema.num_columns(); cid++) {
+ DCHECK_GE(dst_block.mutable_columns()[cid]->size(), n);
+ dst_block.mutable_columns()[cid]->pop_back(n);
+ }
+ }
+}
+
+void BlockAggregator::append_or_merge_row(vectorized::MutableBlock& dst_block,
+ vectorized::Block* src_block, int64_t rid,
+ BitmapValue& skip_bitmap, bool have_delete_sign) {
+ if (have_delete_sign) {
+ // remove all the previous batched rows
+ remove_last_n_rows(dst_block, _state.rows);
+ _state.rows = 0;
+ _state.has_row_with_delete_sign = true;
+
+ append_one_row(dst_block, src_block, rid);
+ } else {
+ if (_state.should_merge()) {
+ merge_one_row(dst_block, src_block, rid, skip_bitmap);
+ } else {
+ append_one_row(dst_block, src_block, rid);
+ }
+ }
+};
+
+Status BlockAggregator::aggregate_rows(
+ vectorized::MutableBlock& output_block, vectorized::Block* block, int64_t start,
+ int64_t end, std::string key, std::vector<BitmapValue>* skip_bitmaps,
+ const signed char* delete_signs, vectorized::IOlapColumnDataAccessor* seq_column,
+ const std::vector<RowsetSharedPtr>& specified_rowsets,
+ std::vector<std::unique_ptr<SegmentCacheHandle>>& segment_caches) {
+ VLOG_DEBUG << fmt::format("merge rows in range=[{}-{})", start, end);
+ if (end - start == 1) {
+ output_block.add_row(block, start);
+ VLOG_DEBUG << fmt::format("append a row directly, rid={}", start);
+ return Status::OK();
+ }
+
+ auto seq_col_unique_id = _tablet_schema.column(_tablet_schema.sequence_col_idx()).unique_id();
+ auto delete_sign_col_unique_id =
+ _tablet_schema.column(_tablet_schema.delete_sign_idx()).unique_id();
+
+ _state.reset();
+
+ RowLocation loc;
+ RowsetSharedPtr rowset;
+ std::string previous_encoded_seq_value {};
+ Status st = _writer._tablet->lookup_row_key(
+ key, &_tablet_schema, false, specified_rowsets, &loc, _writer._mow_context->max_version,
+ segment_caches, &rowset, true, &previous_encoded_seq_value);
+ int64_t pos = start;
+ bool is_expected_st = (st.is<ErrorCode::KEY_NOT_FOUND>() || st.ok());
+ DCHECK(is_expected_st || st.is<ErrorCode::MEM_LIMIT_EXCEEDED>())
+ << "[BlockAggregator::aggregate_rows] unexpected error status while lookup_row_key:"
+ << st;
+ if (!is_expected_st) {
+ return st;
+ }
+
+ std::string cur_seq_val;
+ if (st.ok()) {
+ for (int64_t i {start}; i < end; i++) {
+ auto& skip_bitmap = skip_bitmaps->at(i);
+ bool row_has_sequence_col = (!skip_bitmap.contains(seq_col_unique_id));
+ // Discard all the rows whose seq value is smaller than previous_encoded_seq_value.
+ if (row_has_sequence_col) {
+ std::string seq_val {};
+ _writer._encode_seq_column(seq_column, i, &seq_val);
+ if (Slice {seq_val}.compare(Slice {previous_encoded_seq_value}) < 0) {
+ continue;
+ }
+ cur_seq_val = std::move(seq_val);
+ pos = i;
+ break;
+ }
+ cur_seq_val = std::move(previous_encoded_seq_value);
+ pos = i;
+ break;
+ }
+ } else {
+ pos = start;
+ auto& skip_bitmap = skip_bitmaps->at(pos);
+ bool row_has_sequence_col = (!skip_bitmap.contains(seq_col_unique_id));
+ if (row_has_sequence_col) {
+ std::string seq_val {};
+ // for rows that don't specify seqeunce col, seq_val will be encoded to minial value
+ _writer._encode_seq_column(seq_column, pos, &seq_val);
+ cur_seq_val = std::move(seq_val);
+ } else {
+ cur_seq_val.clear();
+ RETURN_IF_ERROR(_writer._generate_encoded_default_seq_value(
+ _tablet_schema, *_writer._opts.rowset_ctx->partial_update_info, &cur_seq_val));
+ }
+ }
+ VLOG_DEBUG << fmt::format("start pos={}, output_block.rows()={}", pos, output_block.rows());
+
+ for (int64_t rid {pos}; rid < end; rid++) {
+ auto& skip_bitmap = skip_bitmaps->at(rid);
+ bool row_has_sequence_col = (!skip_bitmap.contains(seq_col_unique_id));
+ bool have_delete_sign =
+ (!skip_bitmap.contains(delete_sign_col_unique_id) && delete_signs[rid] != 0);
+ if (!row_has_sequence_col) {
+ append_or_merge_row(output_block, block, rid, skip_bitmap, have_delete_sign);
+ } else {
+ std::string seq_val {};
+ _writer._encode_seq_column(seq_column, rid, &seq_val);
+ if (Slice {seq_val}.compare(Slice {cur_seq_val}) >= 0) {
+ append_or_merge_row(output_block, block, rid, skip_bitmap, have_delete_sign);
+ cur_seq_val = std::move(seq_val);
+ } else {
+ VLOG_DEBUG << fmt::format(
+ "skip rid={} becasue its seq value is lower than the previous", rid);
+ }
+ }
+ }
+ return Status::OK();
+};
+
+Status BlockAggregator::aggregate_for_sequence_column(
+ vectorized::Block* block, size_t num_rows,
+ const std::vector<vectorized::IOlapColumnDataAccessor*>& key_columns,
+ vectorized::IOlapColumnDataAccessor* seq_column,
+ const std::vector<RowsetSharedPtr>& specified_rowsets,
+ std::vector<std::unique_ptr<SegmentCacheHandle>>& segment_caches) {
+ DCHECK_EQ(block->columns(), _tablet_schema.num_columns());
+ VLOG_DEBUG << fmt::format("BlockAggregator::aggregate_for_sequence_column enter: block:{}\n",
+ block->dump_data());
+ // the process logic here is the same as MemTable::_aggregate_for_flexible_partial_update_without_seq_col()
+ // after this function, there will be at most 2 rows for a specified key
+ std::vector<BitmapValue>* skip_bitmaps =
+ &(assert_cast<vectorized::ColumnBitmap*>(
+ block->get_by_position(_tablet_schema.skip_bitmap_col_idx())
+ .column->assume_mutable()
+ .get())
+ ->get_data());
+ const auto* delete_signs = BaseTablet::get_delete_sign_column_data(*block, num_rows);
+
+ auto filtered_block = _tablet_schema.create_block();
+ vectorized::MutableBlock output_block =
+ vectorized::MutableBlock::build_mutable_block(&filtered_block);
+
+ int same_key_rows {0};
+ std::string previous_key {};
+ for (size_t block_pos {0}; block_pos < num_rows; block_pos++) {
+ std::string key = _writer._full_encode_keys(key_columns, block_pos);
+ if (block_pos > 0 && previous_key == key) {
+ same_key_rows++;
+ } else {
+ if (same_key_rows > 0) {
+ RETURN_IF_ERROR(aggregate_rows(output_block, block, block_pos - same_key_rows,
+ block_pos, std::move(previous_key), skip_bitmaps,
+ delete_signs, seq_column, specified_rowsets,
+ segment_caches));
+ }
+ same_key_rows = 1;
+ }
+ previous_key = std::move(key);
+ }
+ if (same_key_rows > 0) {
+ RETURN_IF_ERROR(aggregate_rows(output_block, block, num_rows - same_key_rows, num_rows,
+ std::move(previous_key), skip_bitmaps, delete_signs,
+ seq_column, specified_rowsets, segment_caches));
+ }
+
+ block->swap(output_block.to_block());
+ VLOG_DEBUG << fmt::format(
+ "BlockAggregator::aggregate_for_sequence_column after: data.block:{}\n",
+ block->dump_data());
+ return Status::OK();
+}
+
+Status BlockAggregator::fill_sequence_column(vectorized::Block* block, size_t num_rows,
+ const FixedReadPlan& read_plan,
+ std::vector<BitmapValue>& skip_bitmaps) {
+ DCHECK(_tablet_schema.has_sequence_col());
+ std::vector<uint32_t> cids {static_cast<uint32_t>(_tablet_schema.sequence_col_idx())};
+ auto seq_col_unique_id = _tablet_schema.column(_tablet_schema.sequence_col_idx()).unique_id();
+
+ auto seq_col_block = _tablet_schema.create_block_by_cids(cids);
+ auto tmp_block = _tablet_schema.create_block_by_cids(cids);
+ std::map<uint32_t, uint32_t> read_index;
+ RETURN_IF_ERROR(read_plan.read_columns_by_plan(_tablet_schema, cids, _writer._rsid_to_rowset,
+ seq_col_block, &read_index));
+
+ auto new_seq_col_ptr = tmp_block.get_by_position(0).column->assume_mutable();
+ const auto& old_seq_col_ptr = *seq_col_block.get_by_position(0).column;
+ const auto& cur_seq_col_ptr = *block->get_by_position(_tablet_schema.sequence_col_idx()).column;
+ for (size_t block_pos {0}; block_pos < num_rows; block_pos++) {
+ if (read_index.contains(block_pos)) {
+ new_seq_col_ptr->insert_from(old_seq_col_ptr, read_index[block_pos]);
+ skip_bitmaps[block_pos].remove(seq_col_unique_id);
+ } else {
+ new_seq_col_ptr->insert_from(cur_seq_col_ptr, block_pos);
+ }
+ }
+ block->replace_by_position(_tablet_schema.sequence_col_idx(), std::move(new_seq_col_ptr));
+ return Status::OK();
+}
+
+Status BlockAggregator::aggregate_for_insert_after_delete(
Review Comment:
warning: function 'aggregate_for_insert_after_delete' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
Status BlockAggregator::aggregate_for_insert_after_delete(
^
```
<details>
<summary>Additional context</summary>
**be/src/olap/partial_update_info.cpp:905:** 96 lines including whitespace and comments (threshold 80)
```cpp
Status BlockAggregator::aggregate_for_insert_after_delete(
^
```
</details>
##########
be/src/olap/partial_update_info.cpp:
##########
@@ -658,4 +664,433 @@ Status FlexibleReadPlan::fill_non_primary_key_columns_for_row_store(
return Status::OK();
}
+BlockAggregator::BlockAggregator(segment_v2::VerticalSegmentWriter& vertical_segment_writer)
+ : _writer(vertical_segment_writer), _tablet_schema(*_writer._tablet_schema) {}
+
+void BlockAggregator::merge_one_row(vectorized::MutableBlock& dst_block,
+ vectorized::Block* src_block, int64_t rid,
+ BitmapValue& skip_bitmap) {
+ for (size_t cid {_tablet_schema.num_key_columns()}; cid < _tablet_schema.num_columns(); cid++) {
+ if (cid == _tablet_schema.skip_bitmap_col_idx()) {
+ auto& cur_skip_bitmap =
+ assert_cast<vectorized::ColumnBitmap*>(dst_block.mutable_columns()[cid].get())
+ ->get_data()
+ .back();
+ const auto& new_row_skip_bitmap =
+ assert_cast<vectorized::ColumnBitmap*>(
+ src_block->get_by_position(cid).column->assume_mutable().get())
+ ->get_data()[rid];
+ cur_skip_bitmap &= new_row_skip_bitmap;
+ continue;
+ }
+ if (!skip_bitmap.contains(_tablet_schema.column(cid).unique_id())) {
+ dst_block.mutable_columns()[cid]->pop_back(1);
+ dst_block.mutable_columns()[cid]->insert_from(*src_block->get_by_position(cid).column,
+ rid);
+ }
+ }
+ VLOG_DEBUG << fmt::format("merge a row, after merge, output_block.rows()={}, state: {}",
+ dst_block.rows(), _state.to_string());
+}
+
+void BlockAggregator::append_one_row(vectorized::MutableBlock& dst_block,
+ vectorized::Block* src_block, int64_t rid) {
+ dst_block.add_row(src_block, rid);
+ _state.rows++;
+ VLOG_DEBUG << fmt::format("append a new row, after append, output_block.rows()={}, state: {}",
+ dst_block.rows(), _state.to_string());
+}
+
+void BlockAggregator::remove_last_n_rows(vectorized::MutableBlock& dst_block, int n) {
+ if (n > 0) {
+ for (size_t cid {0}; cid < _tablet_schema.num_columns(); cid++) {
+ DCHECK_GE(dst_block.mutable_columns()[cid]->size(), n);
+ dst_block.mutable_columns()[cid]->pop_back(n);
+ }
+ }
+}
+
+void BlockAggregator::append_or_merge_row(vectorized::MutableBlock& dst_block,
+ vectorized::Block* src_block, int64_t rid,
+ BitmapValue& skip_bitmap, bool have_delete_sign) {
+ if (have_delete_sign) {
+ // remove all the previous batched rows
+ remove_last_n_rows(dst_block, _state.rows);
+ _state.rows = 0;
+ _state.has_row_with_delete_sign = true;
+
+ append_one_row(dst_block, src_block, rid);
+ } else {
+ if (_state.should_merge()) {
+ merge_one_row(dst_block, src_block, rid, skip_bitmap);
+ } else {
+ append_one_row(dst_block, src_block, rid);
+ }
+ }
+};
+
+Status BlockAggregator::aggregate_rows(
Review Comment:
warning: function 'aggregate_rows' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
Status BlockAggregator::aggregate_rows(
^
```
<details>
<summary>Additional context</summary>
**be/src/olap/partial_update_info.cpp:731:** 86 lines including whitespace and comments (threshold 80)
```cpp
Status BlockAggregator::aggregate_rows(
^
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix incorrect result when there are insert after delete for same key in one load [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2443455635
TeamCity be ut coverage result:
Function Coverage: 37.46% (9722/25954)
Line Coverage: 28.70% (80617/280861)
Region Coverage: 28.13% (41687/148196)
Branch Coverage: 24.69% (21172/85758)
Coverage Report: http://coverage.selectdb-in.cc/coverage/2bde941d7f0c31261a07d9bdc7873b8f1521d792_2bde941d7f0c31261a07d9bdc7873b8f1521d792/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2493488019
<details>
<summary>ClickBench: <b>Total hot run time: 33.55 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 61a7fd52e74446b3f27eccd533514d31d5b431bf, data reload: false
query1 0.03 0.03 0.03
query2 0.09 0.04 0.05
query3 0.23 0.06 0.06
query4 1.66 0.09 0.09
query5 0.41 0.41 0.40
query6 1.17 0.68 0.66
query7 0.01 0.02 0.02
query8 0.06 0.04 0.05
query9 0.56 0.50 0.50
query10 0.56 0.54 0.57
query11 0.16 0.12 0.12
query12 0.15 0.13 0.12
query13 0.61 0.61 0.61
query14 2.69 2.88 2.74
query15 0.91 0.84 0.83
query16 0.37 0.35 0.39
query17 1.04 1.02 0.99
query18 0.19 0.19 0.20
query19 1.86 1.83 2.04
query20 0.01 0.01 0.01
query21 15.35 0.67 0.65
query22 4.31 6.83 2.26
query23 18.28 1.43 1.27
query24 2.05 0.25 0.21
query25 0.16 0.08 0.09
query26 0.27 0.17 0.18
query27 0.08 0.08 0.08
query28 13.29 1.18 1.15
query29 12.67 3.33 3.34
query30 0.24 0.06 0.06
query31 2.87 0.40 0.40
query32 3.26 0.50 0.50
query33 2.99 3.13 3.04
query34 17.34 4.52 4.54
query35 4.57 4.64 4.54
query36 0.66 0.48 0.49
query37 0.20 0.16 0.17
query38 0.15 0.15 0.15
query39 0.05 0.05 0.05
query40 0.17 0.14 0.13
query41 0.10 0.06 0.06
query42 0.06 0.06 0.05
query43 0.05 0.04 0.05
Total cold run time: 111.94 s
Total hot run time: 33.55 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2497256678
TeamCity be ut coverage result:
Function Coverage: 38.28% (9980/26070)
Line Coverage: 29.38% (83513/284209)
Region Coverage: 28.55% (42984/150582)
Branch Coverage: 25.13% (21837/86892)
Coverage Report: http://coverage.selectdb-in.cc/coverage/996d2c977562785418ae9b0265c994d07cca4e51_996d2c977562785418ae9b0265c994d07cca4e51/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2635579570
run cloud_p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2664850365
<details>
<summary>TPC-DS: <b>Total hot run time: 182969 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f94d45e93c29681112808b185d09bb4ea8965344, data reload: false
query1 983 396 378 378
query2 6521 1848 1786 1786
query3 6792 208 207 207
query4 26940 23408 23288 23288
query5 4348 676 531 531
query6 301 215 186 186
query7 4608 491 301 301
query8 303 248 236 236
query9 8618 2522 2498 2498
query10 491 301 255 255
query11 15593 15131 14976 14976
query12 150 104 109 104
query13 1664 521 370 370
query14 8681 6240 6544 6240
query15 208 183 171 171
query16 7129 618 456 456
query17 1184 707 539 539
query18 1940 392 289 289
query19 187 189 158 158
query20 127 118 114 114
query21 210 126 107 107
query22 4111 4239 4030 4030
query23 33893 32660 32829 32660
query24 7664 2412 2408 2408
query25 558 471 401 401
query26 1226 273 166 166
query27 2474 461 334 334
query28 4317 2398 2399 2398
query29 773 563 444 444
query30 240 186 160 160
query31 961 849 822 822
query32 79 66 67 66
query33 579 362 310 310
query34 794 812 505 505
query35 784 870 740 740
query36 976 984 897 897
query37 118 104 74 74
query38 4018 4144 4041 4041
query39 1459 1397 1494 1397
query40 214 114 102 102
query41 52 51 48 48
query42 118 102 102 102
query43 497 501 473 473
query44 1274 855 789 789
query45 174 167 160 160
query46 848 1029 647 647
query47 1746 1758 1691 1691
query48 386 411 286 286
query49 788 511 412 412
query50 676 724 401 401
query51 4169 4210 4146 4146
query52 103 103 92 92
query53 221 259 187 187
query54 477 470 407 407
query55 81 78 78 78
query56 261 263 245 245
query57 1127 1160 1054 1054
query58 244 254 235 235
query59 2419 2543 2491 2491
query60 278 278 265 265
query61 124 115 117 115
query62 799 712 637 637
query63 236 203 193 193
query64 4355 1000 697 697
query65 3281 3149 3205 3149
query66 1148 401 317 317
query67 15696 15675 15391 15391
query68 7901 755 497 497
query69 467 299 264 264
query70 1201 1114 1108 1108
query71 414 298 269 269
query72 5722 3581 3723 3581
query73 722 692 355 355
query74 9145 9078 8973 8973
query75 3350 3155 2670 2670
query76 3277 1164 730 730
query77 520 366 333 333
query78 9860 10071 9303 9303
query79 2455 803 605 605
query80 621 528 460 460
query81 504 277 236 236
query82 674 127 91 91
query83 170 173 156 156
query84 251 92 70 70
query85 793 344 341 341
query86 331 306 302 302
query87 4396 4376 4328 4328
query88 3406 2209 2210 2209
query89 385 320 279 279
query90 1889 189 210 189
query91 139 142 109 109
query92 75 62 57 57
query93 1145 1029 566 566
query94 712 407 285 285
query95 358 274 254 254
query96 479 554 268 268
query97 2788 2834 2730 2730
query98 220 208 199 199
query99 1684 1414 1299 1299
Total cold run time: 269848 ms
Total hot run time: 182969 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix incorrect result when there are insert after delete for same key in one load [doris]
Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2425423364
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix incorrect result when there are insert after delete for same key in one load [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2425526699
<details>
<summary>TPC-H: <b>Total hot run time: 41768 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ae261fd99369ad3a45789fe25ed021eca52ebba7, data reload: false
------ Round 1 ----------------------------------
q1 17858 7772 7409 7409
q2 2035 304 284 284
q3 11810 1086 1186 1086
q4 10571 902 927 902
q5 7782 3127 3091 3091
q6 236 153 148 148
q7 1018 612 609 609
q8 9358 1954 1930 1930
q9 6578 6526 6471 6471
q10 7042 2464 2452 2452
q11 442 238 242 238
q12 417 219 222 219
q13 17773 3006 3055 3006
q14 236 206 217 206
q15 580 516 526 516
q16 638 582 586 582
q17 999 557 631 557
q18 7386 6693 6730 6693
q19 1362 1073 975 975
q20 478 188 188 188
q21 3995 3199 3360 3199
q22 1117 1007 1025 1007
Total cold run time: 109711 ms
Total hot run time: 41768 ms
----- Round 2, with runtime_filter_mode=off -----
q1 7340 7255 7268 7255
q2 342 237 259 237
q3 3041 2981 3019 2981
q4 2085 1822 1801 1801
q5 5806 5811 5809 5809
q6 231 146 146 146
q7 2271 1833 1802 1802
q8 3395 3465 3539 3465
q9 9003 8935 8958 8935
q10 3603 3560 3545 3545
q11 585 502 482 482
q12 860 633 594 594
q13 10445 3193 3204 3193
q14 304 272 271 271
q15 583 527 530 527
q16 709 624 634 624
q17 1876 1628 1641 1628
q18 8431 7833 7522 7522
q19 1730 1515 1426 1426
q20 2165 1890 1853 1853
q21 5555 5373 5447 5373
q22 1151 1062 1100 1062
Total cold run time: 71511 ms
Total hot run time: 60531 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix incorrect result when there are insert after delete for same key in one load [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2425535300
TeamCity be ut coverage result:
Function Coverage: 37.46% (9710/25920)
Line Coverage: 28.75% (80640/280466)
Region Coverage: 28.17% (41714/148078)
Branch Coverage: 24.75% (21203/85672)
Coverage Report: http://coverage.selectdb-in.cc/coverage/ae261fd99369ad3a45789fe25ed021eca52ebba7_ae261fd99369ad3a45789fe25ed021eca52ebba7/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix incorrect result when there are insert after delete for same key in one load [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring commented on code in PR #41701:
URL: https://github.com/apache/doris/pull/41701#discussion_r1798671513
##########
be/src/olap/memtable.cpp:
##########
@@ -403,7 +405,7 @@ void MemTable::_sort_one_column(std::vector<RowInBlock*>& row_in_blocks, Tie& ti
}
}
-template <bool is_final>
+template <bool is_final, bool clear_arena>
Review Comment:
Please add comment for clear_arena.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix incorrect result when there are insert after delete for same key in one load [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #41701:
URL: https://github.com/apache/doris/pull/41701#discussion_r1804615505
##########
be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:
##########
@@ -831,98 +885,249 @@ Status VerticalSegmentWriter::_generate_flexible_read_plan(
Status VerticalSegmentWriter::_merge_rows_for_sequence_column(
Review Comment:
warning: function '_merge_rows_for_sequence_column' has cognitive complexity of 76 (threshold 50) [readability-function-cognitive-complexity]
```cpp
Status VerticalSegmentWriter::_merge_rows_for_sequence_column(
^
```
<details>
<summary>Additional context</summary>
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:905:** nesting level increased to 1
```cpp
auto append_row = [&](int64_t rid, BitmapValue& skip_bitmap, bool have_delete_sign) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:906:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (have_delete_sign) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:919:** +1, nesting level increased to 2
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:920:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if ((cur_rows == 1 && !have_rows_with_delete_sign) || cur_rows == 2) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:920:** +1
```cpp
if ((cur_rows == 1 && !have_rows_with_delete_sign) || cur_rows == 2) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:920:** +1
```cpp
if ((cur_rows == 1 && !have_rows_with_delete_sign) || cur_rows == 2) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:943:** +1, nesting level increased to 3
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:954:** nesting level increased to 1
```cpp
auto aggregate_rows = [&](std::string key, int64_t start, int64_t end) -> Status {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:956:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (end - start == 1) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:971:** +1
```cpp
DCHECK(st.is<KEY_NOT_FOUND>() || st.ok());
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:974:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (st.ok()) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:975:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
for (int64_t i {start}; i < end; i++) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:981:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
if (!row_has_sequence_col) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:986:** +1, nesting level increased to 4
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:989:** +5, including nesting penalty of 4, nesting level increased to 5
```cpp
if (Slice {seq_val}.compare(Slice {previous_encoded_seq_value}) >= 0) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:997:** +1, nesting level increased to 2
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1002:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (row_has_sequence_col) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1007:** +1, nesting level increased to 3
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1009:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
RETURN_IF_ERROR(_generate_encoded_default_seq_value(
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1009:** +5, including nesting penalty of 4, nesting level increased to 5
```cpp
RETURN_IF_ERROR(_generate_encoded_default_seq_value(
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1016:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
for (int64_t rid {pos + 1}; rid < end; rid++) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1020:** +1
```cpp
(!skip_bitmap.contains(delete_sign_col_unique_id) && delete_signs[rid] != 0);
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1021:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (!row_has_sequence_col) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1023:** +1, nesting level increased to 3
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1026:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
if (Slice {seq_val}.compare(Slice {cur_seq_val}) >= 0) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1029:** +1, nesting level increased to 4
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1040:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
for (size_t block_pos {0}; block_pos < data.num_rows; block_pos++) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1042:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (block_pos > 0 && previous_key == key) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1042:** +1
```cpp
if (block_pos > 0 && previous_key == key) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1044:** +1, nesting level increased to 2
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1045:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (same_key_rows > 0) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1046:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
RETURN_IF_ERROR(aggregate_rows(std::move(previous_key), block_pos - same_key_rows,
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1046:** +5, including nesting penalty of 4, nesting level increased to 5
```cpp
RETURN_IF_ERROR(aggregate_rows(std::move(previous_key), block_pos - same_key_rows,
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1053:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
if (same_key_rows > 0) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1054:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(aggregate_rows(std::move(previous_key), data.num_rows - same_key_rows,
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1054:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(aggregate_rows(std::move(previous_key), data.num_rows - same_key_rows,
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix incorrect result when there are insert after delete for same key in one load [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2422342008
<details>
<summary>TPC-H: <b>Total hot run time: 41407 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e1f3877e946fc22d7a55d804f2346266653ae2fd, data reload: false
------ Round 1 ----------------------------------
q1 17582 7470 7305 7305
q2 2063 162 153 153
q3 10574 1159 1150 1150
q4 10235 837 896 837
q5 7737 3130 3112 3112
q6 240 159 151 151
q7 1020 618 599 599
q8 9359 1990 2010 1990
q9 6621 6445 6425 6425
q10 7058 2464 2406 2406
q11 443 250 244 244
q12 416 222 226 222
q13 17805 3055 3005 3005
q14 244 209 216 209
q15 559 516 526 516
q16 641 598 581 581
q17 988 566 529 529
q18 7503 6578 6782 6578
q19 1354 948 1018 948
q20 473 190 187 187
q21 4099 3435 3252 3252
q22 1105 1012 1008 1008
Total cold run time: 108119 ms
Total hot run time: 41407 ms
----- Round 2, with runtime_filter_mode=off -----
q1 7328 7247 7234 7234
q2 331 234 224 224
q3 2965 2862 2821 2821
q4 1973 1758 1727 1727
q5 5557 5523 5553 5523
q6 232 144 151 144
q7 2149 1753 1746 1746
q8 3372 3473 3423 3423
q9 8670 8627 8680 8627
q10 3510 3515 3452 3452
q11 585 492 487 487
q12 787 612 573 573
q13 9710 3021 3068 3021
q14 289 263 265 263
q15 554 521 512 512
q16 681 664 629 629
q17 1836 1642 1599 1599
q18 7885 7446 7329 7329
q19 1689 1493 1554 1493
q20 2067 1850 1846 1846
q21 5435 5326 5249 5249
q22 1125 1030 1026 1026
Total cold run time: 68730 ms
Total hot run time: 58948 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix incorrect result when there are insert after delete for same key in one load [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #41701:
URL: https://github.com/apache/doris/pull/41701#discussion_r1805983678
##########
be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:
##########
@@ -831,97 +878,287 @@
Status VerticalSegmentWriter::_merge_rows_for_sequence_column(
Review Comment:
warning: function '_merge_rows_for_sequence_column' has cognitive complexity of 79 (threshold 50) [readability-function-cognitive-complexity]
```cpp
Status VerticalSegmentWriter::_merge_rows_for_sequence_column(
^
```
<details>
<summary>Additional context</summary>
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:899:** nesting level increased to 1
```cpp
auto append_row = [&](int64_t rid, BitmapValue& skip_bitmap, bool have_delete_sign) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:900:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (have_delete_sign) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:902:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (cur_rows > 0) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:915:** +1, nesting level increased to 2
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:916:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if ((cur_rows == 1 && !have_rows_with_delete_sign) || cur_rows == 2) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:916:** +1
```cpp
if ((cur_rows == 1 && !have_rows_with_delete_sign) || cur_rows == 2) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:916:** +1
```cpp
if ((cur_rows == 1 && !have_rows_with_delete_sign) || cur_rows == 2) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:939:** +1, nesting level increased to 3
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:950:** nesting level increased to 1
```cpp
auto aggregate_rows = [&](std::string key, int64_t start, int64_t end) -> Status {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:952:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (end - start == 1) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:967:** +1
```cpp
DCHECK(st.is<KEY_NOT_FOUND>() || st.ok());
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:970:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (st.ok()) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:971:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
for (int64_t i {start}; i < end; i++) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:977:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
if (!row_has_sequence_col) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:981:** +1, nesting level increased to 4
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:984:** +5, including nesting penalty of 4, nesting level increased to 5
```cpp
if (Slice {seq_val}.compare(Slice {previous_encoded_seq_value}) >= 0) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:991:** +1, nesting level increased to 2
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:995:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (row_has_sequence_col) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1000:** +1, nesting level increased to 3
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1002:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
RETURN_IF_ERROR(_generate_encoded_default_seq_value(
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1002:** +5, including nesting penalty of 4, nesting level increased to 5
```cpp
RETURN_IF_ERROR(_generate_encoded_default_seq_value(
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1009:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
for (int64_t rid {pos}; rid < end; rid++) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1013:** +1
```cpp
(!skip_bitmap.contains(delete_sign_col_unique_id) && delete_signs[rid] != 0);
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1014:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (!row_has_sequence_col) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1016:** +1, nesting level increased to 3
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1019:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
if (Slice {seq_val}.compare(Slice {cur_seq_val}) >= 0) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1022:** +1, nesting level increased to 4
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1033:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
for (size_t block_pos {0}; block_pos < data.num_rows; block_pos++) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1035:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (block_pos > 0 && previous_key == key) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1035:** +1
```cpp
if (block_pos > 0 && previous_key == key) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1037:** +1, nesting level increased to 2
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1038:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (same_key_rows > 0) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1039:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
RETURN_IF_ERROR(aggregate_rows(std::move(previous_key), block_pos - same_key_rows,
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1039:** +5, including nesting penalty of 4, nesting level increased to 5
```cpp
RETURN_IF_ERROR(aggregate_rows(std::move(previous_key), block_pos - same_key_rows,
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1046:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
if (same_key_rows > 0) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1047:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(aggregate_rows(std::move(previous_key), data.num_rows - same_key_rows,
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1047:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(aggregate_rows(std::move(previous_key), data.num_rows - same_key_rows,
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
</details>
##########
be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:
##########
@@ -558,6 +567,32 @@ Status VerticalSegmentWriter::_append_block_with_partial_content(RowsInBlock& da
return Status::OK();
}
+Status VerticalSegmentWriter::_filter_block(RowsInBlock& data,
+ vectorized::MutableColumnPtr filter_column,
+ int duplicate_rows, std::string col_name) {
+ auto num_cols = data.block->columns();
+ auto* block = const_cast<vectorized::Block*>(data.block);
+ block->insert(
+ {std::move(filter_column), std::make_shared<vectorized::DataTypeUInt8>(), col_name});
+ RETURN_IF_ERROR(vectorized::Block::filter_block(block, num_cols, num_cols));
+ DCHECK_EQ(num_cols, data.block->columns());
+ int merged_rows = data.num_rows - block->rows();
+ VLOG_DEBUG << fmt::format(
+ "filter_block_for_flexible_partial_update[{}] after filter: "
+ "data.block:{}\n",
+ col_name, data.block->dump_data());
+ if (duplicate_rows != merged_rows) {
+ auto msg = fmt::format(
+ "filter_block_for_flexible_partial_update {}: duplicate_rows != merged_rows, "
+ "duplicate_keys={}, merged_rows={}, num_rows={}, mutable_block->rows()={}",
+ col_name, duplicate_rows, merged_rows, data.num_rows, block->rows());
+ DCHECK(false) << msg;
+ return Status::InternalError<false>(msg);
+ }
+ data.num_rows = block->rows();
+ return Status::OK();
+}
+
Status VerticalSegmentWriter::_append_block_with_flexible_partial_content(
Review Comment:
warning: function '_append_block_with_flexible_partial_content' has cognitive complexity of 68 (threshold 50) [readability-function-cognitive-complexity]
```cpp
Status VerticalSegmentWriter::_append_block_with_flexible_partial_content(
^
```
<details>
<summary>Additional context</summary>
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:613:** nesting level increased to 1
```cpp
auto get_skip_bitmaps = [&skip_bitmap_col_idx](const vectorized::Block* block) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:631:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
DBUG_EXECUTE_IF("VerticalSegmentWriter._append_block_with_flexible_partial_content.sleep",
^
```
**be/src/util/debug_points.h:36:** expanded from macro 'DBUG_EXECUTE_IF'
```cpp
if (UNLIKELY(config::enable_debug_points)) { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:631:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
DBUG_EXECUTE_IF("VerticalSegmentWriter._append_block_with_flexible_partial_content.sleep",
^
```
**be/src/util/debug_points.h:38:** expanded from macro 'DBUG_EXECUTE_IF'
```cpp
if (dp) { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:640:** nesting level increased to 1
```cpp
[&full_block, &data,
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:656:** nesting level increased to 1
```cpp
auto encode_seq_column = [&data, &schema_has_sequence_col,
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:659:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (schema_has_sequence_col) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:661:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(_olap_data_convertor->set_source_content_with_specifid_column(
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:661:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
RETURN_IF_ERROR(_olap_data_convertor->set_source_content_with_specifid_column(
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:665:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (!status.ok()) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:676:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
RETURN_IF_ERROR(encode_pk_columns(key_columns));
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:676:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(encode_pk_columns(key_columns));
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:683:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
RETURN_IF_ERROR(encode_seq_column(seq_column));
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:683:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(encode_seq_column(seq_column));
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:690:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
if (schema_has_sequence_col) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:691:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(_merge_rows_for_sequence_column(data, skip_bitmaps, key_columns, seq_column,
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:691:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(_merge_rows_for_sequence_column(data, skip_bitmaps, key_columns, seq_column,
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:694:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (origin_rows != data.num_rows) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:697:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(encode_pk_columns(key_columns));
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:697:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
RETURN_IF_ERROR(encode_pk_columns(key_columns));
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:698:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(encode_seq_column(seq_column));
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:698:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
RETURN_IF_ERROR(encode_seq_column(seq_column));
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:707:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
RETURN_IF_ERROR(_merge_rows_for_insert_after_delete(
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:707:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(_merge_rows_for_insert_after_delete(
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:709:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
if (data.num_rows != origin_rows) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:712:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(encode_pk_columns(key_columns));
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:712:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(encode_pk_columns(key_columns));
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:713:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(encode_seq_column(seq_column));
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:713:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(encode_seq_column(seq_column));
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:731:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
RETURN_IF_ERROR(_generate_flexible_read_plan(
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:731:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(_generate_flexible_read_plan(
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:737:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
if (config::enable_merge_on_write_correctness_check) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:745:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
RETURN_IF_ERROR(read_plan.fill_non_primary_key_columns(
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:745:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(read_plan.fill_non_primary_key_columns(
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:779:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
if (_num_rows_written != data.row_pos ||
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:788:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
RETURN_IF_ERROR(_generate_primary_key_index(_key_coders, key_columns, seq_column, data.num_rows,
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:788:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(_generate_primary_key_index(_key_coders, key_columns, seq_column, data.num_rows,
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
</details>
##########
be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:
##########
@@ -831,97 +878,287 @@
Status VerticalSegmentWriter::_merge_rows_for_sequence_column(
RowsInBlock& data, std::vector<BitmapValue>* skip_bitmaps,
const std::vector<vectorized::IOlapColumnDataAccessor*>& key_columns,
- vectorized::IOlapColumnDataAccessor* seq_column,
+ vectorized::IOlapColumnDataAccessor* seq_column, const signed char* delete_signs,
const std::vector<RowsetSharedPtr>& specified_rowsets,
std::vector<std::unique_ptr<SegmentCacheHandle>>& segment_caches) {
VLOG_DEBUG << fmt::format(
"VerticalSegmentWriter::_merge_rows_for_sequence_column enter: data.block:{}\n",
data.block->dump_data());
+ // the process logic here is the same as MemTable::_aggregate_for_flexible_partial_update_without_seq_col()
+ // after this function, there will be at most 2 rows for a specified key
+ auto* block = const_cast<vectorized::Block*>(data.block);
auto seq_col_unique_id = _tablet_schema->column(_tablet_schema->sequence_col_idx()).unique_id();
+ auto delete_sign_col_unique_id =
+ _tablet_schema->column(_tablet_schema->delete_sign_idx()).unique_id();
+
+ auto filtered_block = _tablet_schema->create_block();
+ vectorized::MutableBlock output_block =
+ vectorized::MutableBlock::build_mutable_block(&filtered_block);
+ int cur_rows {0};
+ bool have_rows_with_delete_sign {false};
+
+ auto append_row = [&](int64_t rid, BitmapValue& skip_bitmap, bool have_delete_sign) {
+ if (have_delete_sign) {
+ // remove all the preious batched rows
+ if (cur_rows > 0) {
+ for (size_t cid {0}; cid < _tablet_schema->num_columns(); cid++) {
+ DCHECK_GE(output_block.mutable_columns()[cid]->size(), cur_rows);
+ output_block.mutable_columns()[cid]->pop_back(cur_rows);
+ }
+ }
+ output_block.add_row(block, rid);
+ cur_rows = 1;
+ have_rows_with_delete_sign = true;
+ VLOG_DEBUG << fmt::format(
+ "append a row with delete sign, after append, cur_rows={}, "
+ "output_block.rows()={}",
+ cur_rows, output_block.rows());
+ } else {
+ if ((cur_rows == 1 && !have_rows_with_delete_sign) || cur_rows == 2) {
+ for (size_t cid {0}; cid < _tablet_schema->num_columns(); cid++) {
+ if (cid == _tablet_schema->skip_bitmap_col_idx()) {
+ auto& cur_skip_bitmap = assert_cast<vectorized::ColumnBitmap*>(
+ output_block.mutable_columns()[cid].get())
+ ->get_data()
+ .back();
+ const auto& new_row_skip_bitmap =
+ assert_cast<vectorized::ColumnBitmap*>(
+ block->get_by_position(cid).column->assume_mutable().get())
+ ->get_data()[rid];
+ cur_skip_bitmap &= new_row_skip_bitmap;
+ continue;
+ }
+ if (!skip_bitmap.contains(_tablet_schema->column(cid).unique_id())) {
+ output_block.mutable_columns()[cid]->pop_back(1);
+ output_block.mutable_columns()[cid]->insert_from(
+ *block->get_by_position(cid).column, rid);
+ }
+ }
+ VLOG_DEBUG << fmt::format(
+ "merge a row, after merge, cur_rows={}, output_block.rows()={}", cur_rows,
+ output_block.rows());
+ } else {
+ output_block.add_row(block, rid);
+ cur_rows++;
+ VLOG_DEBUG << fmt::format(
+ "append a new row, after append, cur_rows={}, output_block.rows()={}",
+ cur_rows, output_block.rows());
+ }
+ }
+ };
+
+ // aggregate rows with same keys in block range [start, end)
+ auto aggregate_rows = [&](std::string key, int64_t start, int64_t end) -> Status {
+ VLOG_DEBUG << fmt::format("merge rows in range=({}-{})", start, end);
+ if (end - start == 1) {
+ output_block.add_row(block, start);
+ VLOG_DEBUG << fmt::format("append a row directly, rid={}", start);
+ return Status::OK();
+ }
+ cur_rows = 0;
+ have_rows_with_delete_sign = false;
+
+ RowLocation loc;
+ RowsetSharedPtr rowset;
+ std::string previous_encoded_seq_value {};
+ Status st = _tablet->lookup_row_key(key, _tablet_schema.get(), false, specified_rowsets,
+ &loc, _mow_context->max_version, segment_caches,
+ &rowset, true, &previous_encoded_seq_value);
+ int64_t pos = start;
+ DCHECK(st.is<KEY_NOT_FOUND>() || st.ok());
+
+ std::string cur_seq_val;
+ if (st.ok()) {
+ for (int64_t i {start}; i < end; i++) {
+ auto& skip_bitmap = skip_bitmaps->at(i);
+ bool row_has_sequence_col = (!skip_bitmap.contains(seq_col_unique_id));
+ // Find the first row that has larger sequence value than the existing row's
+ // or the first row that doesn't specify the sequence column.
+ // Discard all the rows before and begin to do aggregation from that row
+ if (!row_has_sequence_col) {
+ cur_seq_val = std::move(previous_encoded_seq_value);
+ pos = i;
+ break;
+ } else {
+ std::string seq_val {};
+ _encode_seq_column(seq_column, i, &seq_val);
+ if (Slice {seq_val}.compare(Slice {previous_encoded_seq_value}) >= 0) {
+ cur_seq_val = std::move(seq_val);
+ pos = i;
+ break;
+ }
+ }
+ }
+ } else {
+ pos = start;
+ auto& skip_bitmap = skip_bitmaps->at(pos);
+ bool row_has_sequence_col = (!skip_bitmap.contains(seq_col_unique_id));
+ if (row_has_sequence_col) {
+ std::string seq_val {};
+ // for rows that don't specify seqeunce col, seq_val will be encoded to minial value
+ _encode_seq_column(seq_column, pos, &seq_val);
+ cur_seq_val = std::move(seq_val);
+ } else {
+ cur_seq_val.clear();
+ RETURN_IF_ERROR(_generate_encoded_default_seq_value(
+ *_tablet_schema, *_opts.rowset_ctx->partial_update_info, &cur_seq_val));
+ }
+ }
+ cur_rows = 0;
+ VLOG_DEBUG << fmt::format("start pos={}, output_block.rows()={}", pos, output_block.rows());
+
+ for (int64_t rid {pos}; rid < end; rid++) {
+ auto& skip_bitmap = skip_bitmaps->at(rid);
+ bool row_has_sequence_col = (!skip_bitmap.contains(seq_col_unique_id));
+ bool have_delete_sign =
+ (!skip_bitmap.contains(delete_sign_col_unique_id) && delete_signs[rid] != 0);
+ if (!row_has_sequence_col) {
+ append_row(rid, skip_bitmap, have_delete_sign);
+ } else {
+ std::string seq_val {};
+ _encode_seq_column(seq_column, rid, &seq_val);
+ if (Slice {seq_val}.compare(Slice {cur_seq_val}) >= 0) {
+ append_row(rid, skip_bitmap, have_delete_sign);
+ cur_seq_val = std::move(seq_val);
+ } else {
+ VLOG_DEBUG << fmt::format(
+ "skip rid={} becasue its seq value is lower than the previous", rid);
+ }
+ }
+ }
+ return Status::OK();
+ };
+
+ int same_key_rows {0};
std::string previous_key {};
- bool previous_has_seq_col {false};
- int duplicate_keys {0};
+ for (size_t block_pos {0}; block_pos < data.num_rows; block_pos++) {
+ std::string key = _full_encode_keys(key_columns, block_pos);
+ if (block_pos > 0 && previous_key == key) {
+ same_key_rows++;
+ } else {
+ if (same_key_rows > 0) {
+ RETURN_IF_ERROR(aggregate_rows(std::move(previous_key), block_pos - same_key_rows,
+ block_pos));
+ }
+ same_key_rows = 1;
+ }
+ previous_key = std::move(key);
+ }
+ if (same_key_rows > 0) {
+ RETURN_IF_ERROR(aggregate_rows(std::move(previous_key), data.num_rows - same_key_rows,
+ data.num_rows));
+ }
+ block->swap(output_block.to_block());
+ data.num_rows = block->rows();
+ VLOG_DEBUG << fmt::format(
+ "VerticalSegmentWriter::_merge_rows_for_sequence_column after: data.block:{}\n",
+ data.block->dump_data());
+ return Status::OK();
+}
+
+Status VerticalSegmentWriter::_merge_rows_for_insert_after_delete(
Review Comment:
warning: function '_merge_rows_for_insert_after_delete' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
Status VerticalSegmentWriter::_merge_rows_for_insert_after_delete(
^
```
<details>
<summary>Additional context</summary>
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1059:** 100 lines including whitespace and comments (threshold 80)
```cpp
Status VerticalSegmentWriter::_merge_rows_for_insert_after_delete(
^
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix incorrect result when there are insert after delete for same key in one load [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2434601939
<details>
<summary>TPC-DS: <b>Total hot run time: 193037 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7a452bc70491546bb7a6e70313f42b0b80430833, data reload: false
query1 959 408 409 408
query2 6247 2047 2032 2032
query3 8674 199 208 199
query4 35134 23772 23816 23772
query5 4735 497 469 469
query6 298 172 164 164
query7 4205 298 294 294
query8 286 235 238 235
query9 9574 2671 2657 2657
query10 483 296 275 275
query11 17936 15421 15342 15342
query12 160 105 108 105
query13 1566 416 419 416
query14 10355 7707 7639 7639
query15 251 170 178 170
query16 7941 512 472 472
query17 1556 593 569 569
query18 1822 305 327 305
query19 377 155 162 155
query20 122 116 117 116
query21 215 113 115 113
query22 4694 4381 4623 4381
query23 35274 34062 34290 34062
query24 11050 2809 2800 2800
query25 599 420 415 415
query26 1067 155 168 155
query27 2347 277 286 277
query28 7431 2458 2403 2403
query29 777 436 427 427
query30 265 162 158 158
query31 1063 811 811 811
query32 100 53 55 53
query33 770 299 324 299
query34 913 532 516 516
query35 1018 895 897 895
query36 1135 947 947 947
query37 148 89 91 89
query38 4410 4214 4230 4214
query39 1479 1408 1426 1408
query40 214 104 99 99
query41 49 51 50 50
query42 120 101 101 101
query43 521 485 473 473
query44 1256 809 804 804
query45 199 168 164 164
query46 1117 687 688 687
query47 1922 1832 1843 1832
query48 424 316 326 316
query49 933 439 439 439
query50 816 387 398 387
query51 7058 6903 6942 6903
query52 103 89 88 88
query53 260 180 187 180
query54 1118 443 439 439
query55 84 75 76 75
query56 292 274 294 274
query57 1336 1191 1144 1144
query58 241 235 240 235
query59 3328 3061 3276 3061
query60 290 262 280 262
query61 103 99 107 99
query62 826 688 658 658
query63 223 188 183 183
query64 4133 635 614 614
query65 3306 3230 3193 3193
query66 973 312 310 310
query67 15946 15517 15577 15517
query68 4928 569 555 555
query69 554 300 287 287
query70 1179 1191 1132 1132
query71 344 274 272 272
query72 7155 4037 4016 4016
query73 773 356 358 356
query74 10085 9241 9027 9027
query75 3441 2712 2712 2712
query76 3163 900 890 890
query77 419 296 295 295
query78 10716 9590 9529 9529
query79 1385 616 600 600
query80 2461 431 456 431
query81 588 240 235 235
query82 748 137 132 132
query83 305 135 131 131
query84 281 67 68 67
query85 1356 294 285 285
query86 346 297 278 278
query87 4907 4861 4733 4733
query88 3527 2206 2131 2131
query89 413 285 287 285
query90 2117 183 186 183
query91 148 110 100 100
query92 67 49 51 49
query93 1102 529 529 529
query94 1025 295 296 295
query95 353 245 243 243
query96 614 288 276 276
query97 2878 2664 2685 2664
query98 217 197 194 194
query99 1583 1321 1432 1321
Total cold run time: 303805 ms
Total hot run time: 193037 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2595593729
TeamCity be ut coverage result:
Function Coverage: 41.50% (10823/26078)
Line Coverage: 31.88% (91443/286814)
Region Coverage: 31.03% (46863/151005)
Branch Coverage: 27.13% (23735/87494)
Coverage Report: http://coverage.selectdb-in.cc/coverage/21ec0541fbd8c9eab4a761c02f69ddb72c4856a6_21ec0541fbd8c9eab4a761c02f69ddb72c4856a6/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2595228789
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2609747073
<details>
<summary>TPC-H: <b>Total hot run time: 32262 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 01f1538e375d5d84cf26481834e2500342d61174, data reload: false
------ Round 1 ----------------------------------
q1 17657 5445 5378 5378
q2 2060 313 169 169
q3 10506 1238 725 725
q4 10214 955 543 543
q5 7661 2382 2140 2140
q6 186 174 133 133
q7 911 750 623 623
q8 9232 1371 1129 1129
q9 5147 4945 4952 4945
q10 6831 2329 1867 1867
q11 463 274 253 253
q12 343 355 213 213
q13 17780 3714 3041 3041
q14 238 238 206 206
q15 527 478 461 461
q16 635 631 589 589
q17 580 866 320 320
q18 7275 6497 6521 6497
q19 2535 942 526 526
q20 299 317 189 189
q21 2830 2135 1995 1995
q22 380 326 320 320
Total cold run time: 104290 ms
Total hot run time: 32262 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5559 5481 5513 5481
q2 244 325 238 238
q3 2277 2694 2322 2322
q4 1419 1825 1400 1400
q5 4359 4714 4668 4668
q6 169 169 130 130
q7 2059 1983 1847 1847
q8 2627 2798 2726 2726
q9 7365 7199 7277 7199
q10 3020 3261 2807 2807
q11 574 502 478 478
q12 659 735 588 588
q13 3539 3958 3317 3317
q14 288 310 306 306
q15 514 479 466 466
q16 659 685 653 653
q17 1233 1762 1279 1279
q18 7723 7433 7334 7334
q19 803 1177 1085 1085
q20 2023 2063 1906 1906
q21 5809 5181 4992 4992
q22 610 621 576 576
Total cold run time: 53532 ms
Total hot run time: 51798 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2609773286
<details>
<summary>TPC-DS: <b>Total hot run time: 191935 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 01f1538e375d5d84cf26481834e2500342d61174, data reload: false
query1 1302 956 939 939
query2 6294 2102 2131 2102
query3 10957 4454 4341 4341
query4 60684 29424 23096 23096
query5 5395 609 451 451
query6 422 193 188 188
query7 5552 522 295 295
query8 323 235 215 215
query9 8370 2619 2605 2605
query10 451 332 246 246
query11 17523 15182 15481 15182
query12 167 110 117 110
query13 1426 553 424 424
query14 11029 7063 7650 7063
query15 207 204 194 194
query16 7219 651 494 494
query17 1139 734 582 582
query18 1736 412 313 313
query19 195 179 162 162
query20 115 107 107 107
query21 211 121 103 103
query22 4372 4825 4525 4525
query23 34077 33430 33359 33359
query24 5461 2326 2317 2317
query25 465 476 403 403
query26 638 275 152 152
query27 1555 480 349 349
query28 3997 2461 2456 2456
query29 533 554 424 424
query30 219 187 154 154
query31 925 865 837 837
query32 73 57 58 57
query33 437 359 293 293
query34 734 858 526 526
query35 814 862 791 791
query36 1026 1087 981 981
query37 123 101 87 87
query38 4306 4346 4215 4215
query39 1495 1439 1429 1429
query40 206 116 105 105
query41 54 50 51 50
query42 127 109 105 105
query43 515 548 517 517
query44 1347 819 814 814
query45 185 182 176 176
query46 892 1094 654 654
query47 1944 1931 1860 1860
query48 387 396 322 322
query49 707 487 401 401
query50 663 685 399 399
query51 4279 4297 4239 4239
query52 104 102 93 93
query53 229 262 188 188
query54 489 523 446 446
query55 84 77 80 77
query56 265 276 264 264
query57 1157 1199 1129 1129
query58 246 234 256 234
query59 3257 3431 3263 3263
query60 286 285 254 254
query61 118 119 118 118
query62 766 721 673 673
query63 223 184 185 184
query64 1277 997 688 688
query65 3186 3164 3187 3164
query66 648 390 304 304
query67 15898 15830 15301 15301
query68 2569 832 543 543
query69 425 308 254 254
query70 1209 1153 1127 1127
query71 349 290 256 256
query72 5062 3912 3853 3853
query73 627 812 370 370
query74 9974 8977 8958 8958
query75 3154 3138 2662 2662
query76 1983 1194 777 777
query77 345 373 272 272
query78 10128 10051 9307 9307
query79 1411 822 598 598
query80 1385 513 433 433
query81 550 274 232 232
query82 762 153 122 122
query83 237 185 186 185
query84 290 90 71 71
query85 773 338 297 297
query86 398 309 308 308
query87 4445 4532 4358 4358
query88 3971 2128 2116 2116
query89 390 325 296 296
query90 1747 188 191 188
query91 132 133 109 109
query92 63 61 53 53
query93 1030 882 538 538
query94 681 406 292 292
query95 338 264 263 263
query96 484 617 280 280
query97 2865 2905 2762 2762
query98 222 198 190 190
query99 1310 1370 1276 1276
Total cold run time: 300815 ms
Total hot run time: 191935 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2490434699
<details>
<summary>TPC-H: <b>Total hot run time: 40351 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2d818f829608fbd86f1492c0f3ea5cf6307e7494, data reload: false
------ Round 1 ----------------------------------
q1 17682 7548 7366 7366
q2 2061 180 173 173
q3 10638 1092 1202 1092
q4 10326 784 703 703
q5 7626 2841 2761 2761
q6 246 154 151 151
q7 982 647 615 615
q8 9255 1901 1969 1901
q9 6679 6449 6424 6424
q10 6954 2352 2375 2352
q11 459 259 259 259
q12 423 214 217 214
q13 17769 3076 3060 3060
q14 264 212 222 212
q15 580 526 502 502
q16 647 593 571 571
q17 1009 628 590 590
q18 7364 6630 6731 6630
q19 1353 1028 1028 1028
q20 489 186 186 186
q21 4001 3244 3352 3244
q22 395 323 317 317
Total cold run time: 107202 ms
Total hot run time: 40351 ms
----- Round 2, with runtime_filter_mode=off -----
q1 7297 7288 7297 7288
q2 337 232 227 227
q3 2921 2911 2945 2911
q4 2031 1785 1827 1785
q5 5739 5744 5720 5720
q6 231 148 150 148
q7 2219 1851 1832 1832
q8 3488 3593 3563 3563
q9 8850 8924 8961 8924
q10 3620 3588 3599 3588
q11 615 528 505 505
q12 847 618 660 618
q13 12306 3336 3261 3261
q14 308 284 291 284
q15 566 534 533 533
q16 697 656 632 632
q17 1864 1692 1596 1596
q18 8428 7908 7648 7648
q19 1753 1699 1575 1575
q20 2105 1895 1916 1895
q21 5649 5482 5497 5482
q22 654 591 599 591
Total cold run time: 72525 ms
Total hot run time: 60606 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring merged PR #41701:
URL: https://github.com/apache/doris/pull/41701
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-3061773881
<details>
<summary>ClickBench: <b>Total hot run time: 30.31 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a5f0e6dc17faa644a55c5770a94ae939582a5fb0, data reload: false
query1 0.04 0.03 0.03
query2 0.11 0.05 0.05
query3 0.28 0.06 0.06
query4 1.60 0.08 0.07
query5 0.43 0.41 0.42
query6 1.16 0.67 0.66
query7 0.03 0.01 0.01
query8 0.06 0.05 0.05
query9 0.63 0.54 0.52
query10 0.57 0.59 0.57
query11 0.27 0.13 0.13
query12 0.26 0.13 0.14
query13 0.66 0.63 0.63
query14 0.80 0.82 0.83
query15 0.97 0.90 0.89
query16 0.39 0.38 0.38
query17 1.06 1.08 1.07
query18 0.25 0.23 0.24
query19 1.99 1.84 1.84
query20 0.01 0.01 0.01
query21 15.37 0.97 0.67
query22 0.92 1.02 0.80
query23 14.69 1.53 0.75
query24 5.45 0.56 0.28
query25 0.16 0.08 0.09
query26 0.56 0.22 0.18
query27 0.09 0.09 0.09
query28 11.05 1.18 0.59
query29 12.54 4.15 3.42
query30 0.29 0.09 0.06
query31 2.82 0.63 0.43
query32 3.24 0.60 0.51
query33 3.08 3.14 3.16
query34 16.92 5.55 4.77
query35 4.82 4.78 4.89
query36 0.65 0.52 0.50
query37 0.19 0.18 0.18
query38 0.19 0.17 0.15
query39 0.05 0.04 0.05
query40 0.20 0.18 0.17
query41 0.10 0.05 0.05
query42 0.06 0.06 0.06
query43 0.06 0.05 0.05
Total cold run time: 105.07 s
Total hot run time: 30.31 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-3060257839
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-3060543562
<details>
<summary>TPC-H: <b>Total hot run time: 33375 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8139e2d8e542a5e69903ac8e087381bed76cc5fe, data reload: false
------ Round 1 ----------------------------------
q1 17574 5285 5120 5120
q2 1929 274 190 190
q3 10375 1304 694 694
q4 10246 1050 517 517
q5 7642 2364 2368 2364
q6 173 158 127 127
q7 911 744 597 597
q8 9328 1308 1162 1162
q9 6856 5021 5112 5021
q10 6880 2463 1959 1959
q11 474 285 283 283
q12 339 344 212 212
q13 17758 3735 3069 3069
q14 230 235 217 217
q15 565 476 461 461
q16 444 431 392 392
q17 611 871 376 376
q18 7523 7197 7160 7160
q19 1760 975 543 543
q20 343 349 221 221
q21 3842 3159 2389 2389
q22 346 326 301 301
Total cold run time: 106149 ms
Total hot run time: 33375 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5115 5157 5090 5090
q2 243 329 219 219
q3 2214 2731 2293 2293
q4 1361 1817 1377 1377
q5 4231 4433 4526 4433
q6 237 178 130 130
q7 2040 1984 1835 1835
q8 2651 2514 2553 2514
q9 7425 7268 7343 7268
q10 3103 3310 2915 2915
q11 591 529 495 495
q12 722 753 847 753
q13 3537 3915 3640 3640
q14 283 291 284 284
q15 514 488 491 488
q16 462 500 444 444
q17 1143 1606 1412 1412
q18 7922 7622 7684 7622
q19 825 803 873 803
q20 2026 2127 1920 1920
q21 4742 4286 4572 4286
q22 592 551 557 551
Total cold run time: 51979 ms
Total hot run time: 50772 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2644994911
run cloud_p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2609486814
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2594975565
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2664945829
TeamCity be ut coverage result:
Function Coverage: 43.78% (11442/26133)
Line Coverage: 33.79% (96389/285240)
Region Coverage: 32.48% (49293/151745)
Branch Coverage: 28.14% (24767/88002)
Coverage Report: http://coverage.selectdb-in.cc/coverage/f94d45e93c29681112808b185d09bb4ea8965344_f94d45e93c29681112808b185d09bb4ea8965344/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2642051059
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-3044670993
<details>
<summary>TPC-DS: <b>Total hot run time: 186925 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 269e246b2133708ac5567562989edaa162f27a63, data reload: false
query1 1021 392 399 392
query2 6537 1660 1674 1660
query3 6742 212 210 210
query4 26650 23912 23382 23382
query5 4332 579 423 423
query6 305 242 215 215
query7 4619 495 292 292
query8 271 218 236 218
query9 8618 2640 2656 2640
query10 477 332 295 295
query11 15186 15063 14861 14861
query12 159 105 103 103
query13 1657 518 417 417
query14 8468 5862 5864 5862
query15 198 198 172 172
query16 7727 435 261 261
query17 1336 743 609 609
query18 2019 409 303 303
query19 196 186 175 175
query20 125 119 119 119
query21 215 122 111 111
query22 4076 4245 4523 4245
query23 34952 33968 34055 33968
query24 8557 2424 2372 2372
query25 530 440 383 383
query26 1228 265 143 143
query27 2747 502 344 344
query28 4265 2132 2124 2124
query29 721 531 422 422
query30 284 217 190 190
query31 907 851 772 772
query32 74 59 61 59
query33 554 357 276 276
query34 774 834 531 531
query35 601 644 540 540
query36 950 980 884 884
query37 103 96 74 74
query38 4167 4170 4084 4084
query39 1467 1416 1583 1416
query40 204 117 103 103
query41 59 51 50 50
query42 121 106 104 104
query43 499 482 456 456
query44 1305 833 825 825
query45 172 166 163 163
query46 857 1026 618 618
query47 1751 1760 1694 1694
query48 374 399 300 300
query49 720 467 407 407
query50 624 685 398 398
query51 4129 4168 4180 4168
query52 103 109 99 99
query53 231 247 181 181
query54 584 561 500 500
query55 82 76 82 76
query56 304 287 276 276
query57 1168 1173 1120 1120
query58 263 260 251 251
query59 2520 2686 2598 2598
query60 331 324 314 314
query61 124 123 121 121
query62 828 719 637 637
query63 222 183 184 183
query64 4315 1165 790 790
query65 4239 4140 4247 4140
query66 1091 413 308 308
query67 15892 15512 15293 15293
query68 8780 901 531 531
query69 522 310 278 278
query70 1200 1080 1056 1056
query71 448 308 298 298
query72 5604 4766 4892 4766
query73 738 626 358 358
query74 8830 9215 9019 9019
query75 4102 3197 2693 2693
query76 3633 1153 718 718
query77 855 370 282 282
query78 11003 11078 10202 10202
query79 2171 860 579 579
query80 632 500 436 436
query81 463 259 220 220
query82 447 127 98 98
query83 282 251 231 231
query84 300 103 85 85
query85 788 363 315 315
query86 332 285 266 266
query87 4476 4444 4329 4329
query88 2908 2306 2292 2292
query89 402 311 299 299
query90 1958 228 210 210
query91 148 137 111 111
query92 75 58 54 54
query93 1163 941 589 589
query94 684 314 199 199
query95 377 295 282 282
query96 502 563 281 281
query97 2718 2783 2664 2664
query98 225 215 202 202
query99 1479 1428 1262 1262
Total cold run time: 275172 ms
Total hot run time: 186925 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-3044360347
<details>
<summary>TPC-H: <b>Total hot run time: 33756 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit bb350103fa66a6cc5409e2a4761ac6cb9a96d5fd, data reload: false
------ Round 1 ----------------------------------
q1 17611 5153 5091 5091
q2 1943 292 193 193
q3 10285 1352 743 743
q4 10221 1061 533 533
q5 7549 2372 2398 2372
q6 186 164 130 130
q7 904 753 603 603
q8 9327 1387 1120 1120
q9 6784 5157 5168 5157
q10 6908 2387 1951 1951
q11 491 313 284 284
q12 348 359 223 223
q13 17756 3742 3199 3199
q14 232 225 227 225
q15 534 482 482 482
q16 434 441 400 400
q17 590 878 368 368
q18 7485 7241 7184 7184
q19 1213 942 585 585
q20 335 374 223 223
q21 3785 3220 2390 2390
q22 367 322 300 300
Total cold run time: 105288 ms
Total hot run time: 33756 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5138 5194 5141 5141
q2 245 321 220 220
q3 2179 2699 2307 2307
q4 1444 1828 1348 1348
q5 4286 4386 4604 4386
q6 221 175 133 133
q7 2041 2028 1840 1840
q8 2692 2575 2634 2575
q9 7428 7280 7231 7231
q10 3138 3354 2887 2887
q11 597 532 493 493
q12 730 811 673 673
q13 3754 3997 3517 3517
q14 286 304 272 272
q15 538 481 464 464
q16 461 496 461 461
q17 1208 1658 1431 1431
q18 7910 7772 7566 7566
q19 866 874 970 874
q20 2065 2109 2000 2000
q21 5057 4759 4483 4483
q22 698 639 579 579
Total cold run time: 52982 ms
Total hot run time: 50881 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-3060831237
run external
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-3061719850
<details>
<summary>TPC-H: <b>Total hot run time: 33716 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a5f0e6dc17faa644a55c5770a94ae939582a5fb0, data reload: false
------ Round 1 ----------------------------------
q1 17591 5261 5198 5198
q2 1930 288 193 193
q3 10414 1281 726 726
q4 10243 1021 537 537
q5 7688 2356 2386 2356
q6 180 167 129 129
q7 891 737 614 614
q8 9319 1306 1141 1141
q9 7024 5181 5145 5145
q10 6883 2371 1990 1990
q11 478 298 271 271
q12 333 345 215 215
q13 17775 3753 3075 3075
q14 221 234 214 214
q15 565 478 490 478
q16 437 423 378 378
q17 584 891 361 361
q18 7971 7303 7256 7256
q19 1211 949 558 558
q20 331 364 218 218
q21 3900 3166 2363 2363
q22 354 317 300 300
Total cold run time: 106323 ms
Total hot run time: 33716 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5150 5489 5011 5011
q2 239 319 222 222
q3 2201 2655 2313 2313
q4 1401 1761 1308 1308
q5 4230 4478 4395 4395
q6 217 167 126 126
q7 2041 1990 1779 1779
q8 2716 2749 2618 2618
q9 7210 7174 7185 7174
q10 3142 3355 2846 2846
q11 629 514 519 514
q12 722 822 649 649
q13 3627 4055 3329 3329
q14 290 304 278 278
q15 524 474 470 470
q16 471 500 467 467
q17 1327 1630 1363 1363
q18 7916 7773 7770 7770
q19 893 893 1094 893
q20 2057 2075 1822 1822
q21 4807 4421 4225 4225
q22 640 579 549 549
Total cold run time: 52450 ms
Total hot run time: 50121 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-3061995076
# BE UT Coverage Report
Increment line coverage `3.87% (26/671)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/a5f0e6dc17faa644a55c5770a94ae939582a5fb0_a5f0e6dc17faa644a55c5770a94ae939582a5fb0/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/a5f0e6dc17faa644a55c5770a94ae939582a5fb0_a5f0e6dc17faa644a55c5770a94ae939582a5fb0/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 57.41% (15613/27195) |
| Line Coverage | 46.11% (139313/302107) |
| Region Coverage | 35.49% (104232/293677) |
| Branch Coverage | 37.97% (45921/120955) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-3060811303
# BE UT Coverage Report
Increment line coverage `3.86% (26/674)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/8139e2d8e542a5e69903ac8e087381bed76cc5fe_8139e2d8e542a5e69903ac8e087381bed76cc5fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/8139e2d8e542a5e69903ac8e087381bed76cc5fe_8139e2d8e542a5e69903ac8e087381bed76cc5fe/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 57.09% (15526/27194) |
| Line Coverage | 45.90% (138660/302071) |
| Region Coverage | 35.30% (103660/293681) |
| Branch Coverage | 37.78% (45696/120943) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-3061756698
<details>
<summary>TPC-DS: <b>Total hot run time: 186267 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a5f0e6dc17faa644a55c5770a94ae939582a5fb0, data reload: false
query1 991 392 411 392
query2 6538 1698 1711 1698
query3 6740 205 211 205
query4 27000 23561 22870 22870
query5 4378 578 428 428
query6 299 212 192 192
query7 4622 484 284 284
query8 292 221 206 206
query9 8623 2648 2652 2648
query10 487 323 270 270
query11 15282 14971 14711 14711
query12 156 107 103 103
query13 1665 531 404 404
query14 8646 6014 6030 6014
query15 203 194 181 181
query16 7766 449 265 265
query17 1308 754 593 593
query18 2005 418 302 302
query19 197 191 173 173
query20 130 111 115 111
query21 216 123 103 103
query22 4300 4171 4005 4005
query23 33975 33115 32932 32932
query24 8467 2349 2367 2349
query25 522 469 391 391
query26 1236 264 146 146
query27 2768 504 334 334
query28 4304 2136 2120 2120
query29 711 545 421 421
query30 288 220 216 216
query31 935 842 761 761
query32 74 60 61 60
query33 545 320 285 285
query34 787 846 517 517
query35 608 637 555 555
query36 927 1013 904 904
query37 110 103 75 75
query38 4120 4092 4089 4089
query39 1490 1411 1416 1411
query40 210 119 102 102
query41 54 53 53 53
query42 120 109 113 109
query43 499 513 488 488
query44 1295 811 811 811
query45 173 171 164 164
query46 851 1025 617 617
query47 1739 1810 1696 1696
query48 393 407 308 308
query49 746 464 398 398
query50 636 699 416 416
query51 5441 5493 5480 5480
query52 109 107 98 98
query53 218 253 185 185
query54 576 578 497 497
query55 80 84 79 79
query56 309 330 268 268
query57 1191 1191 1127 1127
query58 272 255 249 249
query59 2631 2756 2587 2587
query60 320 316 291 291
query61 125 120 120 120
query62 808 713 651 651
query63 221 190 183 183
query64 4337 1257 848 848
query65 4317 4155 4192 4155
query66 1068 416 307 307
query67 15800 15772 15493 15493
query68 8305 880 531 531
query69 505 313 264 264
query70 1206 1118 1078 1078
query71 489 316 291 291
query72 5789 4711 4785 4711
query73 736 601 349 349
query74 9076 8830 8900 8830
query75 3868 3209 2640 2640
query76 3734 1153 732 732
query77 788 378 291 291
query78 10977 11017 10186 10186
query79 1996 905 581 581
query80 588 519 435 435
query81 463 268 217 217
query82 419 132 101 101
query83 285 247 231 231
query84 296 102 87 87
query85 835 368 311 311
query86 329 309 292 292
query87 4400 4494 4224 4224
query88 3031 2262 2253 2253
query89 390 322 285 285
query90 1923 209 199 199
query91 138 138 109 109
query92 73 62 61 61
query93 1163 952 593 593
query94 677 319 196 196
query95 368 283 280 280
query96 484 567 274 274
query97 2690 2738 2658 2658
query98 225 215 203 203
query99 1449 1402 1270 1270
Total cold run time: 275945 ms
Total hot run time: 186267 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-3057166395
<details>
<summary>ClickBench: <b>Total hot run time: 29.3 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 49c2fe8f4ea4a7a466b4a04119035ae5a1a9ceaf, data reload: false
query1 0.04 0.04 0.04
query2 0.08 0.04 0.04
query3 0.24 0.08 0.07
query4 1.61 0.11 0.11
query5 0.45 0.44 0.41
query6 1.18 0.66 0.66
query7 0.03 0.02 0.02
query8 0.04 0.03 0.04
query9 0.61 0.53 0.53
query10 0.57 0.56 0.57
query11 0.16 0.12 0.12
query12 0.15 0.13 0.11
query13 0.63 0.61 0.60
query14 0.80 0.82 0.82
query15 0.92 0.89 0.88
query16 0.41 0.40 0.40
query17 1.08 1.10 1.09
query18 0.24 0.22 0.21
query19 2.20 1.87 1.89
query20 0.02 0.01 0.01
query21 15.35 0.89 0.55
query22 0.74 1.13 0.64
query23 15.06 1.39 0.67
query24 7.82 0.80 0.29
query25 0.31 0.21 0.07
query26 0.68 0.17 0.13
query27 0.06 0.06 0.05
query28 8.59 0.89 0.45
query29 12.53 4.03 3.40
query30 0.25 0.09 0.06
query31 2.84 0.60 0.39
query32 3.25 0.57 0.47
query33 3.08 3.09 3.27
query34 16.05 5.36 4.80
query35 4.81 4.86 4.81
query36 0.72 0.53 0.49
query37 0.09 0.07 0.07
query38 0.05 0.05 0.04
query39 0.03 0.02 0.03
query40 0.17 0.14 0.14
query41 0.08 0.03 0.03
query42 0.04 0.03 0.03
query43 0.04 0.03 0.03
Total cold run time: 104.1 s
Total hot run time: 29.3 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix incorrect result when there are insert after delete for same key in one load [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2443956443
<details>
<summary>ClickBench: <b>Total hot run time: 32.54 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d978d67bc61473d4e3aeadc3ec15eff34e8f7593, data reload: false
query1 0.04 0.04 0.03
query2 0.06 0.03 0.03
query3 0.22 0.06 0.06
query4 1.63 0.10 0.10
query5 0.41 0.41 0.42
query6 1.15 0.66 0.65
query7 0.02 0.02 0.01
query8 0.04 0.03 0.04
query9 0.57 0.51 0.48
query10 0.56 0.55 0.55
query11 0.14 0.11 0.12
query12 0.13 0.11 0.11
query13 0.61 0.61 0.60
query14 2.71 2.82 2.74
query15 0.90 0.84 0.81
query16 0.40 0.36 0.39
query17 1.07 1.01 1.05
query18 0.21 0.19 0.19
query19 1.99 1.88 2.02
query20 0.02 0.01 0.01
query21 15.38 0.59 0.60
query22 2.63 2.49 1.14
query23 17.10 0.90 0.78
query24 2.50 1.79 1.48
query25 0.15 0.15 0.06
query26 0.66 0.13 0.14
query27 0.05 0.03 0.03
query28 9.90 1.11 1.07
query29 12.58 3.36 3.37
query30 0.24 0.06 0.06
query31 2.87 0.38 0.37
query32 3.29 0.46 0.45
query33 3.00 3.02 3.02
query34 17.22 4.48 4.50
query35 4.55 4.57 4.58
query36 0.68 0.49 0.47
query37 0.08 0.06 0.06
query38 0.04 0.03 0.04
query39 0.03 0.02 0.03
query40 0.16 0.13 0.13
query41 0.07 0.02 0.03
query42 0.04 0.02 0.02
query43 0.03 0.03 0.03
Total cold run time: 106.13 s
Total hot run time: 32.54 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix incorrect result when there are insert after delete for same key in one load [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #41701:
URL: https://github.com/apache/doris/pull/41701#discussion_r1820537737
##########
be/src/olap/partial_update_info.cpp:
##########
@@ -658,4 +664,431 @@ Status FlexibleReadPlan::fill_non_primary_key_columns_for_row_store(
return Status::OK();
}
+BlockAggregator::BlockAggregator(segment_v2::VerticalSegmentWriter& vertical_segment_writer)
+ : _writer(vertical_segment_writer), _tablet_schema(*_writer._tablet_schema) {}
+
+void BlockAggregator::merge_one_row(vectorized::MutableBlock& dst_block,
+ vectorized::Block* src_block, int64_t rid,
+ BitmapValue& skip_bitmap) {
+ for (size_t cid {_tablet_schema.num_key_columns()}; cid < _tablet_schema.num_columns(); cid++) {
+ if (cid == _tablet_schema.skip_bitmap_col_idx()) {
+ auto& cur_skip_bitmap =
+ assert_cast<vectorized::ColumnBitmap*>(dst_block.mutable_columns()[cid].get())
+ ->get_data()
+ .back();
+ const auto& new_row_skip_bitmap =
+ assert_cast<vectorized::ColumnBitmap*>(
+ src_block->get_by_position(cid).column->assume_mutable().get())
+ ->get_data()[rid];
+ cur_skip_bitmap &= new_row_skip_bitmap;
+ continue;
+ }
+ if (!skip_bitmap.contains(_tablet_schema.column(cid).unique_id())) {
+ dst_block.mutable_columns()[cid]->pop_back(1);
+ dst_block.mutable_columns()[cid]->insert_from(*src_block->get_by_position(cid).column,
+ rid);
+ }
+ }
+ VLOG_DEBUG << fmt::format("merge a row, after merge, output_block.rows()={}, state: {}",
+ dst_block.rows(), _state.to_string());
+}
+
+void BlockAggregator::append_one_row(vectorized::MutableBlock& dst_block,
+ vectorized::Block* src_block, int64_t rid) {
+ dst_block.add_row(src_block, rid);
+ _state.rows++;
+ VLOG_DEBUG << fmt::format("append a new row, after append, output_block.rows()={}, state: {}",
+ dst_block.rows(), _state.to_string());
+}
+
+void BlockAggregator::remove_last_n_rows(vectorized::MutableBlock& dst_block, int n) {
+ if (n > 0) {
+ for (size_t cid {0}; cid < _tablet_schema.num_columns(); cid++) {
+ DCHECK_GE(dst_block.mutable_columns()[cid]->size(), n);
+ dst_block.mutable_columns()[cid]->pop_back(n);
+ }
+ }
+}
+
+void BlockAggregator::append_or_merge_row(vectorized::MutableBlock& dst_block,
+ vectorized::Block* src_block, int64_t rid,
+ BitmapValue& skip_bitmap, bool have_delete_sign) {
+ if (have_delete_sign) {
+ // remove all the previous batched rows
+ remove_last_n_rows(dst_block, _state.rows);
+ _state.rows = 0;
+ _state.has_row_with_delete_sign = true;
+
+ append_one_row(dst_block, src_block, rid);
+ } else {
+ if (_state.should_merge()) {
+ merge_one_row(dst_block, src_block, rid, skip_bitmap);
+ } else {
+ append_one_row(dst_block, src_block, rid);
+ }
+ }
+};
+
+Status BlockAggregator::aggregate_rows(
Review Comment:
warning: function 'aggregate_rows' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
Status BlockAggregator::aggregate_rows(
^
```
<details>
<summary>Additional context</summary>
**be/src/olap/partial_update_info.cpp:731:** 84 lines including whitespace and comments (threshold 80)
```cpp
Status BlockAggregator::aggregate_rows(
^
```
</details>
##########
be/src/olap/partial_update_info.cpp:
##########
@@ -658,4 +664,431 @@
return Status::OK();
}
+BlockAggregator::BlockAggregator(segment_v2::VerticalSegmentWriter& vertical_segment_writer)
+ : _writer(vertical_segment_writer), _tablet_schema(*_writer._tablet_schema) {}
+
+void BlockAggregator::merge_one_row(vectorized::MutableBlock& dst_block,
+ vectorized::Block* src_block, int64_t rid,
+ BitmapValue& skip_bitmap) {
+ for (size_t cid {_tablet_schema.num_key_columns()}; cid < _tablet_schema.num_columns(); cid++) {
+ if (cid == _tablet_schema.skip_bitmap_col_idx()) {
+ auto& cur_skip_bitmap =
+ assert_cast<vectorized::ColumnBitmap*>(dst_block.mutable_columns()[cid].get())
+ ->get_data()
+ .back();
+ const auto& new_row_skip_bitmap =
+ assert_cast<vectorized::ColumnBitmap*>(
+ src_block->get_by_position(cid).column->assume_mutable().get())
+ ->get_data()[rid];
+ cur_skip_bitmap &= new_row_skip_bitmap;
+ continue;
+ }
+ if (!skip_bitmap.contains(_tablet_schema.column(cid).unique_id())) {
+ dst_block.mutable_columns()[cid]->pop_back(1);
+ dst_block.mutable_columns()[cid]->insert_from(*src_block->get_by_position(cid).column,
+ rid);
+ }
+ }
+ VLOG_DEBUG << fmt::format("merge a row, after merge, output_block.rows()={}, state: {}",
+ dst_block.rows(), _state.to_string());
+}
+
+void BlockAggregator::append_one_row(vectorized::MutableBlock& dst_block,
+ vectorized::Block* src_block, int64_t rid) {
+ dst_block.add_row(src_block, rid);
+ _state.rows++;
+ VLOG_DEBUG << fmt::format("append a new row, after append, output_block.rows()={}, state: {}",
+ dst_block.rows(), _state.to_string());
+}
+
+void BlockAggregator::remove_last_n_rows(vectorized::MutableBlock& dst_block, int n) {
+ if (n > 0) {
+ for (size_t cid {0}; cid < _tablet_schema.num_columns(); cid++) {
+ DCHECK_GE(dst_block.mutable_columns()[cid]->size(), n);
+ dst_block.mutable_columns()[cid]->pop_back(n);
+ }
+ }
+}
+
+void BlockAggregator::append_or_merge_row(vectorized::MutableBlock& dst_block,
+ vectorized::Block* src_block, int64_t rid,
+ BitmapValue& skip_bitmap, bool have_delete_sign) {
+ if (have_delete_sign) {
+ // remove all the previous batched rows
+ remove_last_n_rows(dst_block, _state.rows);
+ _state.rows = 0;
+ _state.has_row_with_delete_sign = true;
+
+ append_one_row(dst_block, src_block, rid);
+ } else {
+ if (_state.should_merge()) {
+ merge_one_row(dst_block, src_block, rid, skip_bitmap);
+ } else {
+ append_one_row(dst_block, src_block, rid);
+ }
+ }
+};
+
+Status BlockAggregator::aggregate_rows(
+ vectorized::MutableBlock& output_block, vectorized::Block* block, int64_t start,
+ int64_t end, std::string key, std::vector<BitmapValue>* skip_bitmaps,
+ const signed char* delete_signs, vectorized::IOlapColumnDataAccessor* seq_column,
+ const std::vector<RowsetSharedPtr>& specified_rowsets,
+ std::vector<std::unique_ptr<SegmentCacheHandle>>& segment_caches) {
+ VLOG_DEBUG << fmt::format("merge rows in range=[{}-{})", start, end);
+ if (end - start == 1) {
+ output_block.add_row(block, start);
+ VLOG_DEBUG << fmt::format("append a row directly, rid={}", start);
+ return Status::OK();
+ }
+
+ auto seq_col_unique_id = _tablet_schema.column(_tablet_schema.sequence_col_idx()).unique_id();
+ auto delete_sign_col_unique_id =
+ _tablet_schema.column(_tablet_schema.delete_sign_idx()).unique_id();
+
+ _state.reset();
+
+ RowLocation loc;
+ RowsetSharedPtr rowset;
+ std::string previous_encoded_seq_value {};
+ Status st = _writer._tablet->lookup_row_key(
+ key, &_tablet_schema, false, specified_rowsets, &loc, _writer._mow_context->max_version,
+ segment_caches, &rowset, true, &previous_encoded_seq_value);
+ int64_t pos = start;
+ bool is_expected_st = (st.is<ErrorCode::KEY_NOT_FOUND>() || st.ok());
+ DCHECK(is_expected_st || st.is<ErrorCode::MEM_LIMIT_EXCEEDED>())
+ << "[BlockAggregator::aggregate_rows] unexpected error status while lookup_row_key:"
+ << st;
+ if (!is_expected_st) {
+ return st;
+ }
+
+ std::string cur_seq_val;
+ if (st.ok()) {
+ for (pos = start; pos < end; pos++) {
+ auto& skip_bitmap = skip_bitmaps->at(pos);
+ bool row_has_sequence_col = (!skip_bitmap.contains(seq_col_unique_id));
+ // Discard all the rows whose seq value is smaller than previous_encoded_seq_value.
+ if (row_has_sequence_col) {
+ std::string seq_val {};
+ _writer._encode_seq_column(seq_column, pos, &seq_val);
+ if (Slice {seq_val}.compare(Slice {previous_encoded_seq_value}) < 0) {
+ continue;
+ }
+ cur_seq_val = std::move(seq_val);
+ break;
+ }
+ cur_seq_val = std::move(previous_encoded_seq_value);
+ break;
+ }
+ } else {
+ pos = start;
+ auto& skip_bitmap = skip_bitmaps->at(pos);
+ bool row_has_sequence_col = (!skip_bitmap.contains(seq_col_unique_id));
+ if (row_has_sequence_col) {
+ std::string seq_val {};
+ // for rows that don't specify seqeunce col, seq_val will be encoded to minial value
+ _writer._encode_seq_column(seq_column, pos, &seq_val);
+ cur_seq_val = std::move(seq_val);
+ } else {
+ cur_seq_val.clear();
+ RETURN_IF_ERROR(_writer._generate_encoded_default_seq_value(
+ _tablet_schema, *_writer._opts.rowset_ctx->partial_update_info, &cur_seq_val));
+ }
+ }
+ VLOG_DEBUG << fmt::format("start pos={}, output_block.rows()={}", pos, output_block.rows());
+
+ for (int64_t rid {pos}; rid < end; rid++) {
+ auto& skip_bitmap = skip_bitmaps->at(rid);
+ bool row_has_sequence_col = (!skip_bitmap.contains(seq_col_unique_id));
+ bool have_delete_sign =
+ (!skip_bitmap.contains(delete_sign_col_unique_id) && delete_signs[rid] != 0);
+ if (!row_has_sequence_col) {
+ append_or_merge_row(output_block, block, rid, skip_bitmap, have_delete_sign);
+ } else {
+ std::string seq_val {};
+ _writer._encode_seq_column(seq_column, rid, &seq_val);
+ if (Slice {seq_val}.compare(Slice {cur_seq_val}) >= 0) {
+ append_or_merge_row(output_block, block, rid, skip_bitmap, have_delete_sign);
+ cur_seq_val = std::move(seq_val);
+ } else {
+ VLOG_DEBUG << fmt::format(
+ "skip rid={} becasue its seq value is lower than the previous", rid);
+ }
+ }
+ }
+ return Status::OK();
+};
+
+Status BlockAggregator::aggregate_for_sequence_column(
+ vectorized::Block* block, size_t num_rows,
+ const std::vector<vectorized::IOlapColumnDataAccessor*>& key_columns,
+ vectorized::IOlapColumnDataAccessor* seq_column,
+ const std::vector<RowsetSharedPtr>& specified_rowsets,
+ std::vector<std::unique_ptr<SegmentCacheHandle>>& segment_caches) {
+ DCHECK_EQ(block->columns(), _tablet_schema.num_columns());
+ VLOG_DEBUG << fmt::format("BlockAggregator::aggregate_for_sequence_column enter: block:{}\n",
+ block->dump_data());
+ // the process logic here is the same as MemTable::_aggregate_for_flexible_partial_update_without_seq_col()
+ // after this function, there will be at most 2 rows for a specified key
+ std::vector<BitmapValue>* skip_bitmaps =
+ &(assert_cast<vectorized::ColumnBitmap*>(
+ block->get_by_position(_tablet_schema.skip_bitmap_col_idx())
+ .column->assume_mutable()
+ .get())
+ ->get_data());
+ const auto* delete_signs = BaseTablet::get_delete_sign_column_data(*block, num_rows);
+
+ auto filtered_block = _tablet_schema.create_block();
+ vectorized::MutableBlock output_block =
+ vectorized::MutableBlock::build_mutable_block(&filtered_block);
+
+ int same_key_rows {0};
+ std::string previous_key {};
+ for (size_t block_pos {0}; block_pos < num_rows; block_pos++) {
+ std::string key = _writer._full_encode_keys(key_columns, block_pos);
+ if (block_pos > 0 && previous_key == key) {
+ same_key_rows++;
+ } else {
+ if (same_key_rows > 0) {
+ RETURN_IF_ERROR(aggregate_rows(output_block, block, block_pos - same_key_rows,
+ block_pos, std::move(previous_key), skip_bitmaps,
+ delete_signs, seq_column, specified_rowsets,
+ segment_caches));
+ }
+ same_key_rows = 1;
+ }
+ previous_key = std::move(key);
+ }
+ if (same_key_rows > 0) {
+ RETURN_IF_ERROR(aggregate_rows(output_block, block, num_rows - same_key_rows, num_rows,
+ std::move(previous_key), skip_bitmaps, delete_signs,
+ seq_column, specified_rowsets, segment_caches));
+ }
+
+ block->swap(output_block.to_block());
+ VLOG_DEBUG << fmt::format(
+ "BlockAggregator::aggregate_for_sequence_column after: data.block:{}\n",
+ block->dump_data());
+ return Status::OK();
+}
+
+Status BlockAggregator::fill_sequence_column(vectorized::Block* block, size_t num_rows,
+ const FixedReadPlan& read_plan,
+ std::vector<BitmapValue>& skip_bitmaps) {
+ DCHECK(_tablet_schema.has_sequence_col());
+ std::vector<uint32_t> cids {static_cast<uint32_t>(_tablet_schema.sequence_col_idx())};
+ auto seq_col_unique_id = _tablet_schema.column(_tablet_schema.sequence_col_idx()).unique_id();
+
+ auto seq_col_block = _tablet_schema.create_block_by_cids(cids);
+ auto tmp_block = _tablet_schema.create_block_by_cids(cids);
+ std::map<uint32_t, uint32_t> read_index;
+ RETURN_IF_ERROR(read_plan.read_columns_by_plan(_tablet_schema, cids, _writer._rsid_to_rowset,
+ seq_col_block, &read_index));
+
+ auto new_seq_col_ptr = tmp_block.get_by_position(0).column->assume_mutable();
+ const auto& old_seq_col_ptr = *seq_col_block.get_by_position(0).column;
+ const auto& cur_seq_col_ptr = *block->get_by_position(_tablet_schema.sequence_col_idx()).column;
+ for (size_t block_pos {0}; block_pos < num_rows; block_pos++) {
+ if (read_index.contains(block_pos)) {
+ new_seq_col_ptr->insert_from(old_seq_col_ptr, read_index[block_pos]);
+ skip_bitmaps[block_pos].remove(seq_col_unique_id);
+ } else {
+ new_seq_col_ptr->insert_from(cur_seq_col_ptr, block_pos);
+ }
+ }
+ block->replace_by_position(_tablet_schema.sequence_col_idx(), std::move(new_seq_col_ptr));
+ return Status::OK();
+}
+
+Status BlockAggregator::aggregate_for_insert_after_delete(
Review Comment:
warning: function 'aggregate_for_insert_after_delete' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
Status BlockAggregator::aggregate_for_insert_after_delete(
^
```
<details>
<summary>Additional context</summary>
**be/src/olap/partial_update_info.cpp:903:** 96 lines including whitespace and comments (threshold 80)
```cpp
Status BlockAggregator::aggregate_for_insert_after_delete(
^
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix incorrect result when there are insert after delete for same key in one load [doris]
Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2422262346
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix incorrect result when there are insert after delete for same key in one load [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2422375976
TeamCity be ut coverage result:
Function Coverage: 37.46% (9712/25924)
Line Coverage: 28.76% (80654/280481)
Region Coverage: 28.17% (41721/148079)
Branch Coverage: 24.76% (21209/85674)
Coverage Report: http://coverage.selectdb-in.cc/coverage/e1f3877e946fc22d7a55d804f2346266653ae2fd_e1f3877e946fc22d7a55d804f2346266653ae2fd/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2475367875
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2556329313
<details>
<summary>TPC-DS: <b>Total hot run time: 196531 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c9f2159f3b4ce84a1b67858cefd8b3ceced09e37, data reload: false
query1 1322 984 936 936
query2 6229 2372 2255 2255
query3 10989 4812 4766 4766
query4 33456 23302 23480 23302
query5 4870 483 474 474
query6 300 201 189 189
query7 3995 310 317 310
query8 303 241 244 241
query9 9602 2743 2766 2743
query10 505 242 242 242
query11 18012 15007 15118 15007
query12 151 105 102 102
query13 1575 431 408 408
query14 10413 6714 7340 6714
query15 258 189 192 189
query16 8099 485 474 474
query17 1556 665 602 602
query18 2224 320 334 320
query19 370 167 163 163
query20 133 120 119 119
query21 217 113 108 108
query22 4708 4612 4509 4509
query23 35055 33969 33733 33733
query24 10663 2541 2625 2541
query25 598 388 395 388
query26 716 162 157 157
query27 2115 353 348 348
query28 6733 2543 2484 2484
query29 757 416 423 416
query30 232 146 151 146
query31 1070 851 840 840
query32 101 55 56 55
query33 776 293 293 293
query34 1155 529 526 526
query35 905 787 783 783
query36 1123 947 962 947
query37 120 82 72 72
query38 4365 4314 4421 4314
query39 1501 1468 1459 1459
query40 202 104 103 103
query41 45 44 47 44
query42 122 108 102 102
query43 544 497 502 497
query44 1328 859 873 859
query45 187 170 172 170
query46 1180 765 752 752
query47 2022 1927 1917 1917
query48 425 334 344 334
query49 909 388 395 388
query50 859 401 405 401
query51 7283 7098 7313 7098
query52 117 93 96 93
query53 260 193 189 189
query54 1192 424 427 424
query55 86 84 80 80
query56 256 254 248 248
query57 1313 1184 1176 1176
query58 240 226 230 226
query59 3525 3416 3037 3037
query60 279 245 271 245
query61 110 102 103 102
query62 870 676 668 668
query63 217 192 196 192
query64 3679 677 634 634
query65 3316 3220 3241 3220
query66 811 307 300 300
query67 15978 15535 15400 15400
query68 5088 582 570 570
query69 486 248 249 248
query70 1206 1148 1118 1118
query71 467 253 253 253
query72 6979 4174 4346 4174
query73 779 368 368 368
query74 10053 8887 9056 8887
query75 3652 2656 2690 2656
query76 3600 1223 1054 1054
query77 569 278 271 271
query78 10236 9398 9344 9344
query79 2771 614 633 614
query80 1209 421 479 421
query81 522 232 231 231
query82 1155 117 122 117
query83 190 155 156 155
query84 286 67 71 67
query85 1298 305 295 295
query86 454 300 287 287
query87 4516 4464 4515 4464
query88 4026 2241 2194 2194
query89 430 294 296 294
query90 2055 197 194 194
query91 138 103 109 103
query92 62 54 55 54
query93 2865 573 568 568
query94 838 285 291 285
query95 358 254 254 254
query96 636 281 284 281
query97 2868 2613 2629 2613
query98 218 214 192 192
query99 1646 1340 1331 1331
Total cold run time: 306283 ms
Total hot run time: 196531 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix incorrect result when there are insert after delete for same key in one load [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2408552483
<details>
<summary>ClickBench: <b>Total hot run time: 33.62 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 02cb8af6bdf15fd2a892f3a48fa8c8b4446761a7, data reload: false
query1 0.04 0.03 0.03
query2 0.07 0.03 0.03
query3 0.23 0.06 0.06
query4 1.65 0.10 0.10
query5 0.52 0.52 0.50
query6 1.14 0.73 0.73
query7 0.02 0.01 0.02
query8 0.03 0.03 0.03
query9 0.56 0.51 0.49
query10 0.55 0.55 0.56
query11 0.14 0.11 0.11
query12 0.13 0.11 0.11
query13 0.62 0.61 0.60
query14 2.86 2.71 2.88
query15 0.90 0.83 0.83
query16 0.38 0.39 0.38
query17 1.00 1.08 1.02
query18 0.19 0.20 0.20
query19 1.96 1.87 2.00
query20 0.02 0.01 0.00
query21 15.35 0.59 0.60
query22 3.09 1.82 1.83
query23 17.28 1.01 1.04
query24 3.43 1.69 2.09
query25 0.31 0.18 0.24
query26 0.60 0.14 0.14
query27 0.05 0.05 0.03
query28 9.16 1.10 1.08
query29 12.58 3.25 3.16
query30 0.24 0.06 0.06
query31 2.90 0.37 0.37
query32 3.28 0.46 0.45
query33 3.04 3.05 3.08
query34 17.08 4.41 4.50
query35 4.53 4.46 4.52
query36 0.66 0.48 0.48
query37 0.08 0.06 0.05
query38 0.05 0.03 0.04
query39 0.03 0.02 0.02
query40 0.16 0.12 0.13
query41 0.07 0.02 0.02
query42 0.03 0.02 0.02
query43 0.04 0.02 0.03
Total cold run time: 107.05 s
Total hot run time: 33.62 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix incorrect result when there are insert after delete for same key in one load [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #41701:
URL: https://github.com/apache/doris/pull/41701#discussion_r1796914425
##########
be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:
##########
@@ -831,15 +885,18 @@ Status VerticalSegmentWriter::_generate_flexible_read_plan(
Status VerticalSegmentWriter::_merge_rows_for_sequence_column(
Review Comment:
warning: function '_merge_rows_for_sequence_column' has cognitive complexity of 76 (threshold 50) [readability-function-cognitive-complexity]
```cpp
Status VerticalSegmentWriter::_merge_rows_for_sequence_column(
^
```
<details>
<summary>Additional context</summary>
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:905:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
RETURN_IF_ERROR(_generate_encoded_default_seq_value(
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:905:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(_generate_encoded_default_seq_value(
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:916:** nesting level increased to 1
```cpp
auto get_idx = [](bool with_seq_col, bool has_delete_sign) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:917:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (!with_seq_col) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:918:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (has_delete_sign) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:920:** +1, nesting level increased to 3
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:923:** +1, nesting level increased to 2
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:924:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (has_delete_sign) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:926:** +1, nesting level increased to 3
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:932:** nesting level increased to 1
```cpp
auto find_rows_to_filter = [&](const std::string& key) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:935:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (has_row_with_seq_col && has_row_without_seq_col) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:935:** +1
```cpp
if (has_row_with_seq_col && has_row_without_seq_col) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:946:** +1
```cpp
DCHECK(st.is<KEY_NOT_FOUND>() || st.ok());
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:948:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (st.is<KEY_NOT_FOUND>()) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:950:** +1, nesting level increased to 3
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:955:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (batched_rows[2] != -1) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:959:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (batched_rows[3] != -1) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:964:** nesting level increased to 3
```cpp
auto remove_rows_without_seq = [&]() {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:965:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
if (batched_rows[0] != -1) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:970:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
if (batched_rows[1] != -1) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:978:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (batched_rows[2] != -1 && batched_rows[3] != -1) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:979:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
if (previous_seq_slice.compare(Slice {row_with_delete_sign_encoded_seq_value}) <=
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:982:** +1, nesting level increased to 4
```cpp
} else if (previous_seq_slice.compare(
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:988:** +1, nesting level increased to 4
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:996:** +1, nesting level increased to 3
```cpp
} else if (batched_rows[2] != -1) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:997:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
if (previous_seq_slice.compare(Slice {row_with_delete_sign_encoded_seq_value}) <=
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1000:** +1, nesting level increased to 4
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1005:** +1, nesting level increased to 3
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1006:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
if (previous_seq_slice.compare(Slice {row_with_delete_sign_encoded_seq_value}) <=
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1009:** +1, nesting level increased to 4
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1020:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
for (size_t block_pos = data.row_pos; block_pos < data.row_pos + data.num_rows; block_pos++) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1026:** +1
```cpp
(!skip_bitmap.contains(delete_sign_col_unique_id) && delete_signs[block_pos] != 0);
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1029:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (delta_pos > 0 && previous_key == key) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1029:** +1
```cpp
if (delta_pos > 0 && previous_key == key) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1030:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (!has_same_rows) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1036:** +1, nesting level increased to 2
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1037:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (has_same_rows) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1045:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
if (has_same_rows) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1048:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
if (duplicate_keys > 0) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1049:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(_filter_block_for_flexible_partial_update(
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1049:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(_filter_block_for_flexible_partial_update(
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix incorrect result when there are insert after delete for same key in one load [doris]
Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2408454815
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix incorrect result when there are insert after delete for same key in one load [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring commented on code in PR #41701:
URL: https://github.com/apache/doris/pull/41701#discussion_r1802411209
##########
be/src/olap/memtable.cpp:
##########
@@ -578,6 +546,200 @@ void MemTable::_aggregate() {
}
}
+template <bool is_final>
+void MemTable::_aggregate_for_flexible_partial_update_without_seq_col(
+ const vectorized::ColumnsWithTypeAndName& block_data,
+ vectorized::MutableBlock& mutable_block, std::vector<RowInBlock*>& temp_row_in_blocks) {
+ RowInBlock* prev_row = nullptr;
+ int row_pos = -1;
+ auto& skip_bitmaps = assert_cast<vectorized::ColumnBitmap*>(
+ mutable_block.mutable_columns()[_skip_bitmap_col_idx].get())
+ ->get_data();
+ auto& delete_signs = assert_cast<vectorized::ColumnInt8*>(
+ mutable_block.mutable_columns()[_delete_sign_col_idx].get())
+ ->get_data();
+ RowInBlock* row_with_delete_sign {nullptr};
+ RowInBlock* row_without_delete_sign {nullptr};
+
+ auto finalize_rows = [&]() {
+ if (row_with_delete_sign != nullptr) {
+ temp_row_in_blocks.push_back(row_with_delete_sign);
+ _finalize_one_row<is_final, false>(row_with_delete_sign, block_data, ++row_pos);
+ row_with_delete_sign = nullptr;
+ }
+ if (row_without_delete_sign != nullptr) {
+ temp_row_in_blocks.push_back(row_without_delete_sign);
+ _finalize_one_row<is_final, false>(row_without_delete_sign, block_data, ++row_pos);
+ row_without_delete_sign = nullptr;
+ }
+ _arena->clear();
+ };
+
+ auto add_row = [&](RowInBlock* row, bool with_delete_sign) {
+ if (with_delete_sign) {
+ row_with_delete_sign = row;
+ } else {
+ row_without_delete_sign = row;
+ }
+ };
+ for (RowInBlock* cur_row : _row_in_blocks) {
+ const BitmapValue& skip_bitmap = skip_bitmaps[cur_row->_row_pos];
+ bool cur_row_has_delete_sign = (!skip_bitmap.contains(_delete_sign_col_unique_id) &&
+ delete_signs[cur_row->_row_pos] != 0);
+ prev_row =
+ (row_with_delete_sign == nullptr) ? row_without_delete_sign : row_with_delete_sign;
+ // compare keys, the keys of row_with_delete_sign and row_without_delete_sign is the same,
+ // choose any of them if it's valid
+ if (prev_row != nullptr && (*_vec_row_comparator)(prev_row, cur_row) == 0) {
+ if (cur_row_has_delete_sign) {
+ if (row_without_delete_sign != nullptr) {
+ // if there exits row without delete sign, remove it first
+ _clear_row_agg(row_without_delete_sign);
+ _stat.merged_rows++;
+ row_without_delete_sign = nullptr;
+ }
+ // and then unconditionally replace the previous row
+ prev_row = row_with_delete_sign;
+ } else {
+ prev_row = row_without_delete_sign;
+ }
Review Comment:
logic here is some complicated replated to prev_row.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix incorrect result when there are insert after delete for same key in one load [doris]
Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2406697111
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix incorrect result when there are insert after delete for same key in one load [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #41701:
URL: https://github.com/apache/doris/pull/41701#discussion_r1804693181
##########
be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:
##########
@@ -831,98 +885,250 @@ Status VerticalSegmentWriter::_generate_flexible_read_plan(
Status VerticalSegmentWriter::_merge_rows_for_sequence_column(
Review Comment:
warning: function '_merge_rows_for_sequence_column' has cognitive complexity of 76 (threshold 50) [readability-function-cognitive-complexity]
```cpp
Status VerticalSegmentWriter::_merge_rows_for_sequence_column(
^
```
<details>
<summary>Additional context</summary>
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:906:** nesting level increased to 1
```cpp
auto append_row = [&](int64_t rid, BitmapValue& skip_bitmap, bool have_delete_sign) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:907:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (have_delete_sign) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:920:** +1, nesting level increased to 2
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:921:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if ((cur_rows == 1 && !have_rows_with_delete_sign) || cur_rows == 2) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:921:** +1
```cpp
if ((cur_rows == 1 && !have_rows_with_delete_sign) || cur_rows == 2) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:921:** +1
```cpp
if ((cur_rows == 1 && !have_rows_with_delete_sign) || cur_rows == 2) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:944:** +1, nesting level increased to 3
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:955:** nesting level increased to 1
```cpp
auto aggregate_rows = [&](std::string key, int64_t start, int64_t end) -> Status {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:957:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (end - start == 1) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:972:** +1
```cpp
DCHECK(st.is<KEY_NOT_FOUND>() || st.ok());
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:975:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (st.ok()) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:976:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
for (int64_t i {start}; i < end; i++) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:982:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
if (!row_has_sequence_col) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:987:** +1, nesting level increased to 4
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:990:** +5, including nesting penalty of 4, nesting level increased to 5
```cpp
if (Slice {seq_val}.compare(Slice {previous_encoded_seq_value}) >= 0) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:998:** +1, nesting level increased to 2
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1003:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (row_has_sequence_col) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1008:** +1, nesting level increased to 3
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1010:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
RETURN_IF_ERROR(_generate_encoded_default_seq_value(
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1010:** +5, including nesting penalty of 4, nesting level increased to 5
```cpp
RETURN_IF_ERROR(_generate_encoded_default_seq_value(
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1017:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
for (int64_t rid {pos + 1}; rid < end; rid++) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1021:** +1
```cpp
(!skip_bitmap.contains(delete_sign_col_unique_id) && delete_signs[rid] != 0);
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1022:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (!row_has_sequence_col) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1024:** +1, nesting level increased to 3
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1027:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
if (Slice {seq_val}.compare(Slice {cur_seq_val}) >= 0) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1030:** +1, nesting level increased to 4
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1041:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
for (size_t block_pos {0}; block_pos < data.num_rows; block_pos++) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1043:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (block_pos > 0 && previous_key == key) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1043:** +1
```cpp
if (block_pos > 0 && previous_key == key) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1045:** +1, nesting level increased to 2
```cpp
} else {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1046:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (same_key_rows > 0) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1047:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
RETURN_IF_ERROR(aggregate_rows(std::move(previous_key), block_pos - same_key_rows,
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1047:** +5, including nesting penalty of 4, nesting level increased to 5
```cpp
RETURN_IF_ERROR(aggregate_rows(std::move(previous_key), block_pos - same_key_rows,
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1054:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
if (same_key_rows > 0) {
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1055:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(aggregate_rows(std::move(previous_key), data.num_rows - same_key_rows,
^
```
**be/src/common/status.h:629:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1055:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(aggregate_rows(std::move(previous_key), data.num_rows - same_key_rows,
^
```
**be/src/common/status.h:631:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix incorrect result when there are insert after delete for same key in one load [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2413850972
TeamCity be ut coverage result:
Function Coverage: 37.44% (9708/25932)
Line Coverage: 28.67% (80600/281102)
Region Coverage: 28.14% (41703/148204)
Branch Coverage: 24.71% (21206/85822)
Coverage Report: http://coverage.selectdb-in.cc/coverage/99e53bd3da17aa9c7e310bf4b8d91beb7a81b396_99e53bd3da17aa9c7e310bf4b8d91beb7a81b396/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](flexible partial update) Fix some problems in flexible partial update [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41701:
URL: https://github.com/apache/doris/pull/41701#issuecomment-2561936159
<details>
<summary>TPC-DS: <b>Total hot run time: 189531 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9cc0eab95240250a54bda6aa63dfd1cf1932c83e, data reload: false
query1 990 370 382 370
query2 6514 2297 2297 2297
query3 6700 211 208 208
query4 33971 23660 23373 23373
query5 4342 595 453 453
query6 289 217 192 192
query7 4621 494 313 313
query8 282 238 228 228
query9 9516 2647 2636 2636
query10 471 306 238 238
query11 18330 15396 15169 15169
query12 171 106 101 101
query13 1679 521 397 397
query14 11308 7356 7096 7096
query15 244 203 188 188
query16 8056 606 422 422
query17 1552 750 565 565
query18 2113 389 297 297
query19 219 184 157 157
query20 119 112 118 112
query21 212 128 106 106
query22 4421 4465 4378 4378
query23 34897 33300 33107 33107
query24 6334 2271 2234 2234
query25 506 471 410 410
query26 1192 275 151 151
query27 2010 461 330 330
query28 5162 2440 2384 2384
query29 780 580 498 498
query30 225 184 156 156
query31 972 911 794 794
query32 89 62 59 59
query33 512 338 282 282
query34 742 837 514 514
query35 791 823 757 757
query36 1006 1033 928 928
query37 121 100 74 74
query38 4313 4164 4048 4048
query39 1520 1424 1414 1414
query40 212 110 100 100
query41 48 48 46 46
query42 117 152 102 102
query43 513 545 489 489
query44 1283 809 802 802
query45 176 170 171 170
query46 859 1022 628 628
query47 1893 1899 1850 1850
query48 388 395 315 315
query49 775 473 387 387
query50 604 650 392 392
query51 7134 7125 6983 6983
query52 99 100 91 91
query53 217 251 188 188
query54 460 469 399 399
query55 82 79 77 77
query56 265 250 244 244
query57 1187 1200 1105 1105
query58 239 223 231 223
query59 3143 3250 3055 3055
query60 267 261 248 248
query61 108 109 107 107
query62 851 781 746 746
query63 224 192 189 189
query64 4537 976 647 647
query65 3313 3185 3207 3185
query66 870 410 307 307
query67 15978 15800 15566 15566
query68 9506 752 507 507
query69 482 283 239 239
query70 1189 1148 1123 1123
query71 450 296 255 255
query72 5830 3797 3853 3797
query73 1571 747 352 352
query74 10361 9224 9002 9002
query75 4531 3153 2643 2643
query76 5684 1169 769 769
query77 1020 348 274 274
query78 10022 10360 9411 9411
query79 2626 868 605 605
query80 706 513 481 481
query81 489 264 230 230
query82 354 151 128 128
query83 203 161 143 143
query84 283 87 70 70
query85 743 362 306 306
query86 352 325 304 304
query87 4496 4370 4496 4370
query88 3206 2189 2162 2162
query89 406 334 307 307
query90 1994 186 184 184
query91 138 134 106 106
query92 68 58 86 58
query93 946 866 538 538
query94 635 398 280 280
query95 327 261 247 247
query96 478 603 273 273
query97 2750 2839 2673 2673
query98 227 200 199 199
query99 1667 1531 1418 1418
Total cold run time: 296874 ms
Total hot run time: 189531 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org