You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "Yukang-Lian (via GitHub)" <gi...@apache.org> on 2023/06/16 08:54:22 UTC
[GitHub] [doris] Yukang-Lian opened a new pull request, #20907: [Enhancement](Compaction) Caculate all committed rowsets delete bitmaps when do comapction
Yukang-Lian opened a new pull request, #20907:
URL: https://github.com/apache/doris/pull/20907
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
Here we will calculate all the rowsets delete bitmaps which are committed but not published to reduce the calculation pressure of publish phase.
Step1: collect this tablet's all committed rowsets' delete bitmaps
Step2: calculate all rowsets' delete bitmaps which are published during compaction.
Step3: write back updated delete bitmap and tablet info.
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] Yukang-Lian commented on pull request #20907: [Enhancement](Compaction) Caculate all committed rowsets delete bitmaps when do comapction
Posted by "Yukang-Lian (via GitHub)" <gi...@apache.org>.
Yukang-Lian commented on PR #20907:
URL: https://github.com/apache/doris/pull/20907#issuecomment-1623972601
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20907: [Enhancement](Compaction) Caculate all committed rowsets delete bitmaps when do comapction
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20907:
URL: https://github.com/apache/doris/pull/20907#issuecomment-1624549914
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] Yukang-Lian commented on pull request #20907: [Enhancement](Compaction) Caculate all committed rowsets delete bitmaps when do comapction
Posted by "Yukang-Lian (via GitHub)" <gi...@apache.org>.
Yukang-Lian commented on PR #20907:
URL: https://github.com/apache/doris/pull/20907#issuecomment-1626685544
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #20907: [Enhancement](Compaction) Caculate all committed rowsets delete bitmaps when do comapction
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #20907:
URL: https://github.com/apache/doris/pull/20907#issuecomment-1626744501
TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 49.55 seconds
stream load tsv: 454 seconds loaded 74807831229 Bytes, about 157 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 57 seconds loaded 1101869774 Bytes, about 18 MB/s
stream load parquet: 29 seconds loaded 861443392 Bytes, about 28 MB/s
insert into select: 68.8 seconds inserted 10000000 Rows, about 145K ops/s
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230708040153_clickbench_pr_174788.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] zhannngchen commented on a diff in pull request #20907: [Enhancement](Compaction) Caculate all committed rowsets delete bitmaps when do comapction
Posted by "zhannngchen (via GitHub)" <gi...@apache.org>.
zhannngchen commented on code in PR #20907:
URL: https://github.com/apache/doris/pull/20907#discussion_r1236558263
##########
be/src/olap/compaction.cpp:
##########
@@ -529,8 +532,58 @@ Status Compaction::modify_rowsets(const Merger::Statistics* stats) {
}
}
+ // Here we will calculate all the rowsets delete bitmaps which are committed but not published to reduce the calculation pressure
+ // of publish phase.
+ // All rowsets which need to recalculate have been published so we don't need to acquire lock.
+ // Step1: collect this tablet's all committed rowsets' delete bitmaps
+ TxnManager::txn_tablet_map_t txn_tablet_map {};
+ StorageEngine::instance()->txn_manager()->get_all_tablet_txn_infos_by_tablet(
+ _tablet, txn_tablet_map);
+
+ // Step2: calculate all rowsets' delete bitmaps which are published during compaction.
+ // e.g. before compaction:
+ // 5-5 6-6 published
+ // 7-7 8-8 committed not published
+ // then 5-5 delete bitmap versions are 6, 7(dummy), 8(dummy).
+ // 6-6 delete bitmap versions are 7(dummy), 8(dummy).
+ // when compaction:
+ // 5-5 6-6 7-7 published
+ // 8-8 committed not published
+ // then 5-5 delete bitmap versions are 6, 7, 8(dummy).
+ // 6-6 delete bitmap versions are 7, 8(dummy).
+ // 7-7 doesn't have delete bitmap.
+ // 8-8 has committed, so we want 7-7 delete bitmap version is 8(dummy).
+ // This part is to calculate 7-7's delete bitmap.
+ int64_t cur_max_version = _tablet->max_version().second;
+ for (const auto& it : txn_tablet_map) {
+ for (const auto& tablet_load_it : it.second) {
+ DeltaWriter* delta_writer =
+ StorageEngine::instance()->txn_manager()->get_txn_tablet_delta_writer(
+ it.first.second, _tablet->tablet_id());
+ if (!delta_writer) {
+ continue;
+ }
+ const TabletTxnInfo& tablet_txn_info = tablet_load_it.second;
+ commit_rowset_delete_bitmap.merge(*tablet_txn_info.delete_bitmap);
+ auto beta_rowset = reinterpret_cast<BetaRowset*>(tablet_txn_info.rowset.get());
+ std::vector<segment_v2::SegmentSharedPtr> segments;
+ RETURN_IF_ERROR(beta_rowset->load_segments(&segments));
+ RETURN_IF_ERROR(_tablet->commit_phase_update_delete_bitmap(
Review Comment:
We should not calculate delete bitmap here, which might be quite slow
##########
be/src/olap/compaction.cpp:
##########
@@ -529,8 +532,58 @@ Status Compaction::modify_rowsets(const Merger::Statistics* stats) {
}
}
+ // Here we will calculate all the rowsets delete bitmaps which are committed but not published to reduce the calculation pressure
+ // of publish phase.
+ // All rowsets which need to recalculate have been published so we don't need to acquire lock.
+ // Step1: collect this tablet's all committed rowsets' delete bitmaps
+ TxnManager::txn_tablet_map_t txn_tablet_map {};
Review Comment:
we'd better to move this part after L589, do these works with lock, which can avoid potential consistency issues
##########
be/src/olap/compaction.cpp:
##########
@@ -529,8 +532,58 @@ Status Compaction::modify_rowsets(const Merger::Statistics* stats) {
}
}
+ // Here we will calculate all the rowsets delete bitmaps which are committed but not published to reduce the calculation pressure
+ // of publish phase.
+ // All rowsets which need to recalculate have been published so we don't need to acquire lock.
+ // Step1: collect this tablet's all committed rowsets' delete bitmaps
+ TxnManager::txn_tablet_map_t txn_tablet_map {};
+ StorageEngine::instance()->txn_manager()->get_all_tablet_txn_infos_by_tablet(
+ _tablet, txn_tablet_map);
+
+ // Step2: calculate all rowsets' delete bitmaps which are published during compaction.
+ // e.g. before compaction:
+ // 5-5 6-6 published
+ // 7-7 8-8 committed not published
+ // then 5-5 delete bitmap versions are 6, 7(dummy), 8(dummy).
+ // 6-6 delete bitmap versions are 7(dummy), 8(dummy).
+ // when compaction:
+ // 5-5 6-6 7-7 published
+ // 8-8 committed not published
+ // then 5-5 delete bitmap versions are 6, 7, 8(dummy).
+ // 6-6 delete bitmap versions are 7, 8(dummy).
+ // 7-7 doesn't have delete bitmap.
+ // 8-8 has committed, so we want 7-7 delete bitmap version is 8(dummy).
+ // This part is to calculate 7-7's delete bitmap.
+ int64_t cur_max_version = _tablet->max_version().second;
+ for (const auto& it : txn_tablet_map) {
+ for (const auto& tablet_load_it : it.second) {
+ DeltaWriter* delta_writer =
+ StorageEngine::instance()->txn_manager()->get_txn_tablet_delta_writer(
+ it.first.second, _tablet->tablet_id());
+ if (!delta_writer) {
+ continue;
+ }
+ const TabletTxnInfo& tablet_txn_info = tablet_load_it.second;
+ commit_rowset_delete_bitmap.merge(*tablet_txn_info.delete_bitmap);
+ auto beta_rowset = reinterpret_cast<BetaRowset*>(tablet_txn_info.rowset.get());
+ std::vector<segment_v2::SegmentSharedPtr> segments;
+ RETURN_IF_ERROR(beta_rowset->load_segments(&segments));
+ RETURN_IF_ERROR(_tablet->commit_phase_update_delete_bitmap(
+ tablet_txn_info.rowset, tablet_txn_info.rowset_ids,
+ tablet_txn_info.delete_bitmap, cur_max_version, segments, it.first.second,
+ delta_writer->get_rowset_writer()));
+ // Step3: write back updated delete bitmap and tablet info.
+ StorageEngine::instance()->txn_manager()->set_txn_related_delete_bitmap(
+ it.first.first, it.first.second, _tablet->tablet_id(),
+ _tablet->schema_hash(), _tablet->tablet_uid(), true,
+ tablet_txn_info.delete_bitmap, _tablet->all_rs_id(cur_max_version));
Review Comment:
1. you should update the converted delete bitmap to txn_info
2. add the compacted rowset's id to txn_info.rowset_ids
##########
be/src/olap/compaction.cpp:
##########
@@ -540,7 +593,7 @@ Status Compaction::modify_rowsets(const Merger::Statistics* stats) {
// incremental data.
_tablet->calc_compaction_output_rowset_delete_bitmap(
_input_rowsets, _rowid_conversion, version.second, UINT64_MAX, &missed_rows,
- &location_map, &output_rowset_delete_bitmap);
+ &location_map, &output_rowset_delete_bitmap, commit_rowset_delete_bitmap);
Review Comment:
you should not merge these un-published bitmap to Tablet's bitmap
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20907: [Enhancement](Compaction) Caculate all committed rowsets delete bitmaps when do comapction
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20907:
URL: https://github.com/apache/doris/pull/20907#issuecomment-1595608016
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] Yukang-Lian commented on pull request #20907: [Enhancement](Compaction) Caculate all committed rowsets delete bitmaps when do comapction
Posted by "Yukang-Lian (via GitHub)" <gi...@apache.org>.
Yukang-Lian commented on PR #20907:
URL: https://github.com/apache/doris/pull/20907#issuecomment-1595823295
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] liaoxin01 commented on a diff in pull request #20907: [Enhancement](Compaction) Caculate all committed rowsets delete bitmaps when do comapction
Posted by "liaoxin01 (via GitHub)" <gi...@apache.org>.
liaoxin01 commented on code in PR #20907:
URL: https://github.com/apache/doris/pull/20907#discussion_r1253993465
##########
be/src/olap/txn_manager.cpp:
##########
@@ -632,6 +632,28 @@ void TxnManager::get_all_related_tablets(std::set<TabletInfo>* tablet_infos) {
}
}
+void TxnManager::get_all_commit_tablet_txn_info_by_tablet(
Review Comment:
Just returning references to delete_bitmap and rowset_ids is enough?
##########
be/src/olap/tablet.cpp:
##########
@@ -3240,6 +3240,13 @@ Status Tablet::commit_phase_update_delete_bitmap(
<< ", rowset_ids to del: " << rowset_ids_to_del.size()
<< ", cur max_version: " << cur_version << ", transaction_id: " << txn_id
<< ", cost: " << watch.get_elapse_time_us() << "(us), total rows: " << total_rows;
+
+ for (auto iter = delete_bitmap->delete_bitmap.begin();
Review Comment:
no need to do this conversion?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #20907: [Enhancement](Compaction) Caculate all committed rowsets delete bitmaps when do comapction
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #20907:
URL: https://github.com/apache/doris/pull/20907#issuecomment-1625047753
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 50.85 seconds
stream load tsv: 507 seconds loaded 74807831229 Bytes, about 140 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 87.9 seconds inserted 10000000 Rows, about 113K ops/s
storage size: 17162459196 Bytes
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230707163542_clickbench_pr_174340.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] Yukang-Lian commented on pull request #20907: [Enhancement](Compaction) Caculate all committed rowsets delete bitmaps when do comapction
Posted by "Yukang-Lian (via GitHub)" <gi...@apache.org>.
Yukang-Lian commented on PR #20907:
URL: https://github.com/apache/doris/pull/20907#issuecomment-1594350685
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20907: [Enhancement](Compaction) Caculate all committed rowsets delete bitmaps when do comapction
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20907:
URL: https://github.com/apache/doris/pull/20907#issuecomment-1608720000
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20907: [Enhancement](Compaction) Caculate all committed rowsets delete bitmaps when do comapction
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20907:
URL: https://github.com/apache/doris/pull/20907#issuecomment-1613028693
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20907: [Enhancement](Compaction) Caculate all committed rowsets delete bitmaps when do comapction
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20907:
URL: https://github.com/apache/doris/pull/20907#issuecomment-1612371739
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20907: [Enhancement](Compaction) Caculate all committed rowsets delete bitmaps when do comapction
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20907:
URL: https://github.com/apache/doris/pull/20907#issuecomment-1625179445
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20907: [Enhancement](Compaction) Caculate all committed rowsets delete bitmaps when do comapction
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20907:
URL: https://github.com/apache/doris/pull/20907#issuecomment-1594429610
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] Yukang-Lian commented on pull request #20907: [Enhancement](Compaction) Caculate all committed rowsets delete bitmaps when do comapction
Posted by "Yukang-Lian (via GitHub)" <gi...@apache.org>.
Yukang-Lian commented on PR #20907:
URL: https://github.com/apache/doris/pull/20907#issuecomment-1594654955
run p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20907: [Enhancement](Compaction) Caculate all committed rowsets delete bitmaps when do comapction
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20907:
URL: https://github.com/apache/doris/pull/20907#issuecomment-1612375620
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] Yukang-Lian commented on pull request #20907: [Enhancement](Compaction) Caculate all committed rowsets delete bitmaps when do comapction
Posted by "Yukang-Lian (via GitHub)" <gi...@apache.org>.
Yukang-Lian commented on PR #20907:
URL: https://github.com/apache/doris/pull/20907#issuecomment-1612546847
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #20907: [Enhancement](Compaction) Caculate all committed rowsets delete bitmaps when do comapction
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #20907:
URL: https://github.com/apache/doris/pull/20907#issuecomment-1625217543
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 56.05 seconds
stream load tsv: 507 seconds loaded 74807831229 Bytes, about 140 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 30 seconds loaded 861443392 Bytes, about 27 MB/s
insert into select: 89.2 seconds inserted 10000000 Rows, about 112K ops/s
storage size: 17167907865 Bytes
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230707184029_clickbench_pr_174492.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #20907: [Enhancement](Compaction) Caculate all committed rowsets delete bitmaps when do comapction
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #20907:
URL: https://github.com/apache/doris/pull/20907#issuecomment-1625059475
TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 54.79 seconds
stream load tsv: 457 seconds loaded 74807831229 Bytes, about 156 MB/s
stream load json: 23 seconds loaded 2358488459 Bytes, about 97 MB/s
stream load orc: 57 seconds loaded 1101869774 Bytes, about 18 MB/s
stream load parquet: 29 seconds loaded 861443392 Bytes, about 28 MB/s
insert into select: 67.9 seconds inserted 10000000 Rows, about 147K ops/s
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230707083938_clickbench_pr_174349.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] Yukang-Lian commented on pull request #20907: [Enhancement](Compaction) Caculate all committed rowsets delete bitmaps when do comapction
Posted by "Yukang-Lian (via GitHub)" <gi...@apache.org>.
Yukang-Lian commented on PR #20907:
URL: https://github.com/apache/doris/pull/20907#issuecomment-1625171778
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20907: [Enhancement](Compaction) Caculate all committed rowsets delete bitmaps when do comapction
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20907:
URL: https://github.com/apache/doris/pull/20907#issuecomment-1628012609
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20907: [Enhancement](Compaction) Caculate all committed rowsets delete bitmaps when do comapction
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20907:
URL: https://github.com/apache/doris/pull/20907#issuecomment-1626704073
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] Yukang-Lian commented on pull request #20907: [Enhancement](Compaction) Caculate all committed rowsets delete bitmaps when do comapction
Posted by "Yukang-Lian (via GitHub)" <gi...@apache.org>.
Yukang-Lian commented on PR #20907:
URL: https://github.com/apache/doris/pull/20907#issuecomment-1595639632
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] Yukang-Lian commented on pull request #20907: [Enhancement](Compaction) Caculate all committed rowsets delete bitmaps when do comapction
Posted by "Yukang-Lian (via GitHub)" <gi...@apache.org>.
Yukang-Lian commented on PR #20907:
URL: https://github.com/apache/doris/pull/20907#issuecomment-1624956976
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] zhannngchen merged pull request #20907: [Enhancement](Compaction) Caculate all committed rowsets delete bitmaps when do comapction
Posted by "zhannngchen (via GitHub)" <gi...@apache.org>.
zhannngchen merged PR #20907:
URL: https://github.com/apache/doris/pull/20907
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] Yukang-Lian commented on pull request #20907: [Enhancement](Compaction) Caculate all committed rowsets delete bitmaps when do comapction
Posted by "Yukang-Lian (via GitHub)" <gi...@apache.org>.
Yukang-Lian commented on PR #20907:
URL: https://github.com/apache/doris/pull/20907#issuecomment-1594403321
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20907: [Enhancement](Compaction) Caculate all committed rowsets delete bitmaps when do comapction
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20907:
URL: https://github.com/apache/doris/pull/20907#issuecomment-1594361060
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20907: [Enhancement](Compaction) Caculate all committed rowsets delete bitmaps when do comapction
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20907:
URL: https://github.com/apache/doris/pull/20907#issuecomment-1612366281
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] Yukang-Lian commented on pull request #20907: [Enhancement](Compaction) Caculate all committed rowsets delete bitmaps when do comapction
Posted by "Yukang-Lian (via GitHub)" <gi...@apache.org>.
Yukang-Lian commented on PR #20907:
URL: https://github.com/apache/doris/pull/20907#issuecomment-1612733143
run p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20907: [Enhancement](Compaction) Caculate all committed rowsets delete bitmaps when do comapction
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20907:
URL: https://github.com/apache/doris/pull/20907#issuecomment-1616348537
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20907: [Enhancement](Compaction) Caculate all committed rowsets delete bitmaps when do comapction
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20907:
URL: https://github.com/apache/doris/pull/20907#issuecomment-1609110578
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] Yukang-Lian commented on pull request #20907: [Enhancement](Compaction) Caculate all committed rowsets delete bitmaps when do comapction
Posted by "Yukang-Lian (via GitHub)" <gi...@apache.org>.
Yukang-Lian commented on PR #20907:
URL: https://github.com/apache/doris/pull/20907#issuecomment-1595833899
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] Yukang-Lian commented on pull request #20907: [Enhancement](Compaction) Caculate all committed rowsets delete bitmaps when do comapction
Posted by "Yukang-Lian (via GitHub)" <gi...@apache.org>.
Yukang-Lian commented on PR #20907:
URL: https://github.com/apache/doris/pull/20907#issuecomment-1595827896
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] Yukang-Lian commented on pull request #20907: [Enhancement](Compaction) Caculate all committed rowsets delete bitmaps when do comapction
Posted by "Yukang-Lian (via GitHub)" <gi...@apache.org>.
Yukang-Lian commented on PR #20907:
URL: https://github.com/apache/doris/pull/20907#issuecomment-1595607270
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] zhannngchen commented on a diff in pull request #20907: [Enhancement](Compaction) Caculate all committed rowsets delete bitmaps when do comapction
Posted by "zhannngchen (via GitHub)" <gi...@apache.org>.
zhannngchen commented on code in PR #20907:
URL: https://github.com/apache/doris/pull/20907#discussion_r1244953260
##########
be/src/olap/compaction.cpp:
##########
@@ -574,11 +575,47 @@ Status Compaction::modify_rowsets(const Merger::Statistics* stats) {
RETURN_IF_ERROR(_tablet->check_rowid_conversion(_output_rowset, location_map));
location_map.clear();
+
{
std::lock_guard<std::mutex> wrlock_(_tablet->get_rowset_update_lock());
std::lock_guard<std::shared_mutex> wrlock(_tablet->get_header_lock());
SCOPED_SIMPLE_TRACE_IF_TIMEOUT(TRACE_TABLET_LOCK_THRESHOLD);
+ // Here we will calculate all the rowsets delete bitmaps which are committed but not published to reduce the calculation pressure
+ // of publish phase.
+ // All rowsets which need to recalculate have been published so we don't need to acquire lock.
+ // Step1: collect this tablet's all committed rowsets' delete bitmaps
+ TxnManager::txn_tablet_map_t txn_tablet_map {};
+ StorageEngine::instance()->txn_manager()->get_all_commit_tablet_txn_info_by_tablet(
+ _tablet, txn_tablet_map);
+
+ // Step2: calculate all rowsets' delete bitmaps which are published during compaction.
+ int64_t cur_max_version = _tablet->max_version().second;
+ RowsetIdUnorderedSet rowset_ids = _tablet->all_rs_id(cur_max_version);
+ rowset_ids.insert(_output_rowset->rowset_id());
+ for (const auto& it : txn_tablet_map) {
+ for (const auto& tablet_load_it : it.second) {
+ const TabletTxnInfo& tablet_txn_info = tablet_load_it.second;
+ DeleteBitmap output_delete_bitmap(_tablet->tablet_id());
+ RowLocation src;
+ std::shared_ptr<Rowset> rowset = tablet_txn_info.rowset;
+ src.rowset_id = rowset->rowset_id();
+ for (uint32_t seg_id = 0; seg_id < rowset->num_segments(); ++seg_id) {
+ src.segment_id = seg_id;
+ _tablet->convert_rowid(rowset, *tablet_txn_info.delete_bitmap, src,
Review Comment:
can't use such delete bitmap directly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20907: [Enhancement](Compaction) Caculate all committed rowsets delete bitmaps when do comapction
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20907:
URL: https://github.com/apache/doris/pull/20907#issuecomment-1612997705
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] Yukang-Lian commented on pull request #20907: [Enhancement](Compaction) Caculate all committed rowsets delete bitmaps when do comapction
Posted by "Yukang-Lian (via GitHub)" <gi...@apache.org>.
Yukang-Lian commented on PR #20907:
URL: https://github.com/apache/doris/pull/20907#issuecomment-1624547284
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20907: [Enhancement](Compaction) Caculate all committed rowsets delete bitmaps when do comapction
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20907:
URL: https://github.com/apache/doris/pull/20907#issuecomment-1594363141
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20907: [Enhancement](Compaction) Caculate all committed rowsets delete bitmaps when do comapction
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20907:
URL: https://github.com/apache/doris/pull/20907#issuecomment-1594624898
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20907: [Enhancement](Compaction) Caculate all committed rowsets delete bitmaps when do comapction
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20907:
URL: https://github.com/apache/doris/pull/20907#issuecomment-1594409528
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20907: [Enhancement](Compaction) Caculate all committed rowsets delete bitmaps when do comapction
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20907:
URL: https://github.com/apache/doris/pull/20907#issuecomment-1614286521
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #20907: [Enhancement](Compaction) Caculate all committed rowsets delete bitmaps when do comapction
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #20907:
URL: https://github.com/apache/doris/pull/20907#issuecomment-1626772600
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 55.75 seconds
stream load tsv: 508 seconds loaded 74807831229 Bytes, about 140 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 88.5 seconds inserted 10000000 Rows, about 112K ops/s
storage size: 17162241855 Bytes
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230708121445_clickbench_pr_174790.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #20907: [Enhancement](Compaction) Caculate all committed rowsets delete bitmaps when do comapction
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #20907:
URL: https://github.com/apache/doris/pull/20907#issuecomment-1624579697
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 54.63 seconds
stream load tsv: 512 seconds loaded 74807831229 Bytes, about 139 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 68 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 85.3 seconds inserted 10000000 Rows, about 117K ops/s
storage size: 17162237060 Bytes
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230707105824_clickbench_pr_174064.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20907: [Enhancement](Compaction) Caculate all committed rowsets delete bitmaps when do comapction
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20907:
URL: https://github.com/apache/doris/pull/20907#issuecomment-1620599166
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #20907: [Enhancement](Compaction) Caculate all committed rowsets delete bitmaps when do comapction
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #20907:
URL: https://github.com/apache/doris/pull/20907#issuecomment-1624044434
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 52.16 seconds
stream load tsv: 511 seconds loaded 74807831229 Bytes, about 139 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 30 seconds loaded 861443392 Bytes, about 27 MB/s
insert into select: 87.1 seconds inserted 10000000 Rows, about 114K ops/s
storage size: 17164874736 Bytes
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230707012436_clickbench_pr_173942.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20907: [Enhancement](Compaction) Caculate all committed rowsets delete bitmaps when do comapction
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20907:
URL: https://github.com/apache/doris/pull/20907#issuecomment-1624965620
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #20907: [Enhancement](Compaction) Caculate all committed rowsets delete bitmaps when do comapction
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #20907:
URL: https://github.com/apache/doris/pull/20907#issuecomment-1624573365
TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.85 seconds
stream load tsv: 455 seconds loaded 74807831229 Bytes, about 156 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
stream load parquet: 28 seconds loaded 861443392 Bytes, about 29 MB/s
insert into select: 69.1 seconds inserted 10000000 Rows, about 144K ops/s
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230707024729_clickbench_pr_174066.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] zhannngchen commented on a diff in pull request #20907: [Enhancement](Compaction) Caculate all committed rowsets delete bitmaps when do comapction
Posted by "zhannngchen (via GitHub)" <gi...@apache.org>.
zhannngchen commented on code in PR #20907:
URL: https://github.com/apache/doris/pull/20907#discussion_r1247374314
##########
be/src/olap/compaction.cpp:
##########
@@ -574,16 +577,47 @@ Status Compaction::modify_rowsets(const Merger::Statistics* stats) {
RETURN_IF_ERROR(_tablet->check_rowid_conversion(_output_rowset, location_map));
location_map.clear();
+
{
std::lock_guard<std::mutex> wrlock_(_tablet->get_rowset_update_lock());
std::lock_guard<std::shared_mutex> wrlock(_tablet->get_header_lock());
SCOPED_SIMPLE_TRACE_IF_TIMEOUT(TRACE_TABLET_LOCK_THRESHOLD);
+ // Here we will calculate all the rowsets delete bitmaps which are committed but not published to reduce the calculation pressure
+ // of publish phase.
+ // All rowsets which need to recalculate have been published so we don't need to acquire lock.
+ // Step1: collect this tablet's all committed rowsets' delete bitmaps
+ CommitTabletTxnInfoVec commit_tablet_txn_info_vec {};
+ StorageEngine::instance()->txn_manager()->get_all_commit_tablet_txn_info_by_tablet(
+ _tablet, commit_tablet_txn_info_vec);
+
+ // Step2: calculate all rowsets' delete bitmaps which are published during compaction.
+ int64_t cur_max_version = _tablet->max_version().second;
+ RowsetIdUnorderedSet rowset_ids = _tablet->all_rs_id(cur_max_version);
+ rowset_ids.insert(_output_rowset->rowset_id());
+ for (const auto& it : commit_tablet_txn_info_vec) {
+ DeleteBitmap output_delete_bitmap(_tablet->tablet_id());
+ const std::shared_ptr<Rowset>& rowset = it.rowset;
+ for (uint32_t seg_id = 0; seg_id < rowset->num_segments(); ++seg_id) {
+ _tablet->calc_compaction_output_rowset_delete_bitmap(
+ std::vector<std::shared_ptr<Rowset>>(1, rowset), _rowid_conversion,
Review Comment:
You used wrong input rowset?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20907: [Enhancement](Compaction) Caculate all committed rowsets delete bitmaps when do comapction
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20907:
URL: https://github.com/apache/doris/pull/20907#issuecomment-1614291568
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20907: [Enhancement](Compaction) Caculate all committed rowsets delete bitmaps when do comapction
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20907:
URL: https://github.com/apache/doris/pull/20907#issuecomment-1623964363
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20907: [Enhancement](Compaction) Caculate all committed rowsets delete bitmaps when do comapction
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20907:
URL: https://github.com/apache/doris/pull/20907#issuecomment-1625180115
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20907: [Enhancement](Compaction) Caculate all committed rowsets delete bitmaps when do comapction
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20907:
URL: https://github.com/apache/doris/pull/20907#issuecomment-1620105054
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] Yukang-Lian commented on pull request #20907: [Enhancement](Compaction) Caculate all committed rowsets delete bitmaps when do comapction
Posted by "Yukang-Lian (via GitHub)" <gi...@apache.org>.
Yukang-Lian commented on PR #20907:
URL: https://github.com/apache/doris/pull/20907#issuecomment-1620595475
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] Yukang-Lian commented on pull request #20907: [Enhancement](Compaction) Caculate all committed rowsets delete bitmaps when do comapction
Posted by "Yukang-Lian (via GitHub)" <gi...@apache.org>.
Yukang-Lian commented on PR #20907:
URL: https://github.com/apache/doris/pull/20907#issuecomment-1595820510
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] zhannngchen commented on a diff in pull request #20907: [Enhancement](Compaction) Caculate all committed rowsets delete bitmaps when do comapction
Posted by "zhannngchen (via GitHub)" <gi...@apache.org>.
zhannngchen commented on code in PR #20907:
URL: https://github.com/apache/doris/pull/20907#discussion_r1244787552
##########
be/src/olap/txn_manager.cpp:
##########
@@ -653,6 +653,19 @@ void TxnManager::get_all_related_tablets(std::set<TabletInfo>* tablet_infos) {
}
}
+void TxnManager::get_all_commit_tablet_txn_info_by_tablet(const TabletSharedPtr& tablet,
+ txn_tablet_map_t& txn_tablet_map) {
Review Comment:
seems we just need a list, using a map here is confusing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] Yukang-Lian commented on pull request #20907: [Enhancement](Compaction) Caculate all committed rowsets delete bitmaps when do comapction
Posted by "Yukang-Lian (via GitHub)" <gi...@apache.org>.
Yukang-Lian commented on PR #20907:
URL: https://github.com/apache/doris/pull/20907#issuecomment-1609154941
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20907: [Enhancement](Compaction) Caculate all committed rowsets delete bitmaps when do comapction
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20907:
URL: https://github.com/apache/doris/pull/20907#issuecomment-1595640560
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #20907: [Enhancement](Compaction) Caculate all committed rowsets delete bitmaps when do comapction
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #20907:
URL: https://github.com/apache/doris/pull/20907#issuecomment-1594677675
TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 37.61 seconds
stream load tsv: 457 seconds loaded 74807831229 Bytes, about 156 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 57 seconds loaded 1101869774 Bytes, about 18 MB/s
stream load parquet: 28 seconds loaded 861443392 Bytes, about 29 MB/s
insert into select: 63.9 seconds inserted 10000000 Rows, about 156K ops/s
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230616112909_clickbench_pr_163691.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20907: [Enhancement](Compaction) Caculate all committed rowsets delete bitmaps when do comapction
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20907:
URL: https://github.com/apache/doris/pull/20907#issuecomment-1605610475
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #20907: [Enhancement](Compaction) Caculate all committed rowsets delete bitmaps when do comapction
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #20907:
URL: https://github.com/apache/doris/pull/20907#issuecomment-1624006047
TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 50.77 seconds
stream load tsv: 453 seconds loaded 74807831229 Bytes, about 157 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 57 seconds loaded 1101869774 Bytes, about 18 MB/s
stream load parquet: 28 seconds loaded 861443392 Bytes, about 29 MB/s
insert into select: 70.3 seconds inserted 10000000 Rows, about 142K ops/s
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230706165641_clickbench_pr_173946.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #20907: [Enhancement](Compaction) Caculate all committed rowsets delete bitmaps when do comapction
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #20907:
URL: https://github.com/apache/doris/pull/20907#issuecomment-1625282044
TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 51.85 seconds
stream load tsv: 457 seconds loaded 74807831229 Bytes, about 156 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 57 seconds loaded 1101869774 Bytes, about 18 MB/s
stream load parquet: 29 seconds loaded 861443392 Bytes, about 28 MB/s
insert into select: 71.0 seconds inserted 10000000 Rows, about 140K ops/s
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230707113652_clickbench_pr_174489.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] Yukang-Lian commented on pull request #20907: [Enhancement](Compaction) Caculate all committed rowsets delete bitmaps when do comapction
Posted by "Yukang-Lian (via GitHub)" <gi...@apache.org>.
Yukang-Lian commented on PR #20907:
URL: https://github.com/apache/doris/pull/20907#issuecomment-1613302622
run clickbench
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #20907: [Enhancement](Compaction) Caculate all committed rowsets delete bitmaps when do comapction
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #20907:
URL: https://github.com/apache/doris/pull/20907#issuecomment-1613350361
TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 43.88 seconds
stream load tsv: 459 seconds loaded 74807831229 Bytes, about 155 MB/s
stream load json: 24 seconds loaded 2358488459 Bytes, about 93 MB/s
stream load orc: 57 seconds loaded 1101869774 Bytes, about 18 MB/s
stream load parquet: 29 seconds loaded 861443392 Bytes, about 28 MB/s
insert into select: 70.7 seconds inserted 10000000 Rows, about 141K ops/s
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230629150723_clickbench_pr_170061.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] zhannngchen commented on a diff in pull request #20907: [Enhancement](Compaction) Caculate all committed rowsets delete bitmaps when do comapction
Posted by "zhannngchen (via GitHub)" <gi...@apache.org>.
zhannngchen commented on code in PR #20907:
URL: https://github.com/apache/doris/pull/20907#discussion_r1247374629
##########
be/src/olap/compaction.cpp:
##########
@@ -574,16 +577,47 @@ Status Compaction::modify_rowsets(const Merger::Statistics* stats) {
RETURN_IF_ERROR(_tablet->check_rowid_conversion(_output_rowset, location_map));
location_map.clear();
+
{
std::lock_guard<std::mutex> wrlock_(_tablet->get_rowset_update_lock());
std::lock_guard<std::shared_mutex> wrlock(_tablet->get_header_lock());
SCOPED_SIMPLE_TRACE_IF_TIMEOUT(TRACE_TABLET_LOCK_THRESHOLD);
+ // Here we will calculate all the rowsets delete bitmaps which are committed but not published to reduce the calculation pressure
+ // of publish phase.
+ // All rowsets which need to recalculate have been published so we don't need to acquire lock.
+ // Step1: collect this tablet's all committed rowsets' delete bitmaps
+ CommitTabletTxnInfoVec commit_tablet_txn_info_vec {};
+ StorageEngine::instance()->txn_manager()->get_all_commit_tablet_txn_info_by_tablet(
+ _tablet, commit_tablet_txn_info_vec);
+
+ // Step2: calculate all rowsets' delete bitmaps which are published during compaction.
+ int64_t cur_max_version = _tablet->max_version().second;
+ RowsetIdUnorderedSet rowset_ids = _tablet->all_rs_id(cur_max_version);
Review Comment:
should not replace the rowset_ids with `_tablet->all_rs_id(cur_max_version)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20907: [Enhancement](Compaction) Caculate all committed rowsets delete bitmaps when do comapction
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20907:
URL: https://github.com/apache/doris/pull/20907#issuecomment-1609155334
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20907: [Enhancement](Compaction) Caculate all committed rowsets delete bitmaps when do comapction
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20907:
URL: https://github.com/apache/doris/pull/20907#issuecomment-1594624824
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] Yukang-Lian commented on pull request #20907: [Enhancement](Compaction) Caculate all committed rowsets delete bitmaps when do comapction
Posted by "Yukang-Lian (via GitHub)" <gi...@apache.org>.
Yukang-Lian commented on PR #20907:
URL: https://github.com/apache/doris/pull/20907#issuecomment-1613017976
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20907: [Enhancement](Compaction) Caculate all committed rowsets delete bitmaps when do comapction
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20907:
URL: https://github.com/apache/doris/pull/20907#issuecomment-1612553909
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] Yukang-Lian commented on pull request #20907: [Enhancement](Compaction) Caculate all committed rowsets delete bitmaps when do comapction
Posted by "Yukang-Lian (via GitHub)" <gi...@apache.org>.
Yukang-Lian commented on PR #20907:
URL: https://github.com/apache/doris/pull/20907#issuecomment-1609102595
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] zhannngchen commented on a diff in pull request #20907: [Enhancement](Compaction) Caculate all committed rowsets delete bitmaps when do comapction
Posted by "zhannngchen (via GitHub)" <gi...@apache.org>.
zhannngchen commented on code in PR #20907:
URL: https://github.com/apache/doris/pull/20907#discussion_r1244787127
##########
be/src/olap/tablet.cpp:
##########
@@ -3296,29 +3295,39 @@ void Tablet::calc_compaction_output_rowset_delete_bitmap(
{rowset->rowset_id(), seg_id, end_version},
&subset_map);
// traverse all versions and convert rowid
- for (auto iter = subset_map.delete_bitmap.begin();
- iter != subset_map.delete_bitmap.end(); ++iter) {
- auto cur_version = std::get<2>(iter->first);
- for (auto index = iter->second.begin(); index != iter->second.end(); ++index) {
- src.row_id = *index;
- if (rowid_conversion.get(src, &dst) != 0) {
- VLOG_CRITICAL << "Can't find rowid, may be deleted by the delete_handler, "
- << " src loaction: |" << src.rowset_id << "|"
- << src.segment_id << "|" << src.row_id
- << " version: " << cur_version;
- missed_rows->insert(src);
- continue;
- }
- VLOG_DEBUG << "calc_compaction_output_rowset_delete_bitmap dst location: |"
- << dst.rowset_id << "|" << dst.segment_id << "|" << dst.row_id
- << " src location: |" << src.rowset_id << "|" << src.segment_id
- << "|" << src.row_id << " start version: " << start_version
- << "end version" << end_version;
- (*location_map)[rowset].emplace_back(src, dst);
- output_rowset_delete_bitmap->add({dst.rowset_id, dst.segment_id, cur_version},
- dst.row_id);
- }
+ convert_rowid(rowset, subset_map, src, rowid_conversion, start_version, end_version,
+ missed_rows, location_map, output_rowset_delete_bitmap);
+ }
+ }
+}
+
+void Tablet::convert_rowid(
Review Comment:
This method don't needs so many parameters, we should try our best to make the declaration more clear
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20907: [Enhancement](Compaction) Caculate all committed rowsets delete bitmaps when do comapction
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20907:
URL: https://github.com/apache/doris/pull/20907#issuecomment-1595821734
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20907: [Enhancement](Compaction) Caculate all committed rowsets delete bitmaps when do comapction
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20907:
URL: https://github.com/apache/doris/pull/20907#issuecomment-1595838302
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20907: [Enhancement](Compaction) Caculate all committed rowsets delete bitmaps when do comapction
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20907:
URL: https://github.com/apache/doris/pull/20907#issuecomment-1595824838
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20907: [Enhancement](Compaction) Caculate all committed rowsets delete bitmaps when do comapction
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20907:
URL: https://github.com/apache/doris/pull/20907#issuecomment-1605611767
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20907: [Enhancement](Compaction) Caculate all committed rowsets delete bitmaps when do comapction
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20907:
URL: https://github.com/apache/doris/pull/20907#issuecomment-1605611462
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org