You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2021/09/15 15:18:52 UTC
[GitHub] [incubator-doris] weizuo93 opened a new pull request #6671: [Bug] Set a weight for empty version when calculating compaction score
weizuo93 opened a new pull request #6671:
URL: https://github.com/apache/incubator-doris/pull/6671
## Proposed changes
To avoid that too many rowsets with 0 segment can not be merged when data skew, we should set a weight for empty version when calculating compaction score.
## Types of changes
What types of changes does your code introduce to Doris?
_Put an `x` in the boxes that apply_
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
- [ ] Documentation Update (if none of the other choices apply)
- [ ] Code refactor (Modify the code structure, format the code, etc...)
- [ ] Optimization. Including functional usability improvements and performance improvements.
- [ ] Dependency. Such as changes related to third-party components.
- [ ] Other.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] weizuo93 closed pull request #6671: [Bug] Set a weight for empty version when calculating compaction score
Posted by GitBox <gi...@apache.org>.
weizuo93 closed pull request #6671:
URL: https://github.com/apache/incubator-doris/pull/6671
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] weizuo93 commented on a change in pull request #6671: [Bug] Set a weight for empty version when calculating compaction score
Posted by GitBox <gi...@apache.org>.
weizuo93 commented on a change in pull request #6671:
URL: https://github.com/apache/incubator-doris/pull/6671#discussion_r715658037
##########
File path: be/src/olap/olap_server.cpp
##########
@@ -394,7 +394,7 @@ void StorageEngine::_compaction_tasks_producer_callback() {
for (const auto& tablet : tablets_compaction) {
int64_t permits =
tablet->prepare_compaction_and_calculate_permits(compaction_type, tablet);
- if (permits > 0 && _permit_limiter.request(permits)) {
+ if (permits >= 0 && _permit_limiter.request(permits)) {
Review comment:
> What is (permit == 0) means?
All the rowsets are empty version. The contribution of the empty version to `permit` is 0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request #6671: [Bug] Set a weight for empty version when calculating compaction score
Posted by GitBox <gi...@apache.org>.
morningman commented on a change in pull request #6671:
URL: https://github.com/apache/incubator-doris/pull/6671#discussion_r715440488
##########
File path: be/src/olap/olap_server.cpp
##########
@@ -394,7 +394,7 @@ void StorageEngine::_compaction_tasks_producer_callback() {
for (const auto& tablet : tablets_compaction) {
int64_t permits =
tablet->prepare_compaction_and_calculate_permits(compaction_type, tablet);
- if (permits > 0 && _permit_limiter.request(permits)) {
+ if (permits >= 0 && _permit_limiter.request(permits)) {
Review comment:
What is (permit == 0) means?
##########
File path: be/src/olap/cumulative_compaction_policy.cpp
##########
@@ -390,23 +427,37 @@ int NumBasedCumulativeCompactionPolicy::pick_input_rowsets(
// if we have a sufficient number of segments,
// or have other versions before encountering the delete version, we should process the compaction.
- if (last_delete_version->first == -1 && *compaction_score < min_compaction_score) {
+ if (last_delete_version->first == -1 && (*compaction_score + empty_rowset_num
+ / config::empty_rowset_compaction_score_ratio) < min_compaction_score) {
input_rowsets->clear();
+ *compaction_score = 0;
+ empty_rowset_num = 0;
}
+
+ *compaction_score += empty_rowset_num / config::empty_rowset_compaction_score_ratio;
Review comment:
Move this in front of `if`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org