You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "csun5285 (via GitHub)" <gi...@apache.org> on 2023/07/24 08:45:39 UTC
[GitHub] [doris] csun5285 opened a new pull request, #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
csun5285 opened a new pull request, #22159:
URL: https://github.com/apache/doris/pull/22159
## Proposed changes
Now, there were two global compaction policies available: "size_based" and "time_series". The compaction policy can be specified during table creation.
```
CREATE TABLE IF NOT EXISTS example_db.example_tbl
(
`user_id` LARGEINT NOT NULL,
`date` DATE NOT NULL,
`city` VARCHAR(20),
`age` SMALLINT,
)
DUPLICATE KEY(`user_id`)
DISTRIBUTED BY HASH(`user_id`) BUCKETS 1
PROPERTIES (
"replication_allocation" = "tag.location.default: 1"
"compaction_policy"="time_series",
"time_series_compaction_goal_size_mbyte"="1024"
);
```
Issue Number: close #xxx
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] csun5285 commented on pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
Posted by "csun5285 (via GitHub)" <gi...@apache.org>.
csun5285 commented on PR #22159:
URL: https://github.com/apache/doris/pull/22159#issuecomment-1651138855
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] csun5285 commented on a diff in pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
Posted by "csun5285 (via GitHub)" <gi...@apache.org>.
csun5285 commented on code in PR #22159:
URL: https://github.com/apache/doris/pull/22159#discussion_r1278307841
##########
be/src/olap/cumulative_compaction_time_series_policy.cpp:
##########
@@ -63,13 +63,16 @@ uint32_t TimeSeriesCumulativeCompactionPolicy::calc_cumulative_compaction_score(
return 0;
}
- // Condition 1: the size of input files for compaction meets the requirement of parameter _compaction_goal_size
- if (total_size >= (config::time_series_compaction_goal_size_mbytes * 1024 * 1024)) {
+ // Condition 1: the size of input files for compaction meets the requirement of parameter compaction_goal_size
+ if (total_size >=
+ (tablet->tablet_meta()->tablet_schema()->time_series_compaction_goal_size_mbytes() * 1024 *
Review Comment:
done
##########
be/src/olap/compaction.cpp:
##########
@@ -148,8 +148,10 @@ int64_t Compaction::get_avg_segment_rows() {
// input_rowsets_size is total disk_size of input_rowset, this size is the
// final size after codec and compress, so expect dest segment file size
// in disk is config::vertical_compaction_max_segment_size
- if (config::compaction_policy == CUMULATIVE_TIME_SERIES_POLICY) {
- return (config::time_series_compaction_goal_size_mbytes * 1024 * 1024 * 2) /
+ if (_tablet->tablet_meta()->tablet_schema()->compaction_policy() ==
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] csun5285 commented on a diff in pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
Posted by "csun5285 (via GitHub)" <gi...@apache.org>.
csun5285 commented on code in PR #22159:
URL: https://github.com/apache/doris/pull/22159#discussion_r1278308659
##########
be/src/olap/storage_engine.h:
##########
@@ -462,7 +462,9 @@ class StorageEngine {
std::shared_ptr<StreamLoadRecorder> _stream_load_recorder;
- std::shared_ptr<CumulativeCompactionPolicy> _cumulative_compaction_policy;
+ // we use unordered_map to store all cumulative compaction policy sharded ptr
Review Comment:
BE previously used a single global compaction policy pointer. Now, I want to maintain two compaction policy pointers globally and select different pointers based on the compaction policy for each tablet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] csun5285 commented on pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
Posted by "csun5285 (via GitHub)" <gi...@apache.org>.
csun5285 commented on PR #22159:
URL: https://github.com/apache/doris/pull/22159#issuecomment-1657172716
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #22159:
URL: https://github.com/apache/doris/pull/22159#issuecomment-1651643645
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] csun5285 commented on a diff in pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
Posted by "csun5285 (via GitHub)" <gi...@apache.org>.
csun5285 commented on code in PR #22159:
URL: https://github.com/apache/doris/pull/22159#discussion_r1278308754
##########
be/src/olap/olap_server.cpp:
##########
@@ -989,7 +993,14 @@ Status StorageEngine::submit_compaction_task(TabletSharedPtr tablet, CompactionT
bool force) {
_update_cumulative_compaction_policy();
if (tablet->get_cumulative_compaction_policy() == nullptr) {
- tablet->set_cumulative_compaction_policy(_cumulative_compaction_policy);
Review Comment:
select different pointers based on the compaction policy for each tablet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #22159:
URL: https://github.com/apache/doris/pull/22159#issuecomment-1658140018
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.59 seconds
stream load tsv: 507 seconds loaded 74807831229 Bytes, about 140 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.4 seconds inserted 10000000 Rows, about 340K ops/s
storage size: 17168685360 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #22159:
URL: https://github.com/apache/doris/pull/22159#issuecomment-1658365984
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #22159:
URL: https://github.com/apache/doris/pull/22159#issuecomment-1657105314
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.84 seconds
stream load tsv: 540 seconds loaded 74807831229 Bytes, about 132 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 30.0 seconds inserted 10000000 Rows, about 333K ops/s
storage size: 17162320155 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] csun5285 commented on a diff in pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
Posted by "csun5285 (via GitHub)" <gi...@apache.org>.
csun5285 commented on code in PR #22159:
URL: https://github.com/apache/doris/pull/22159#discussion_r1279004649
##########
fe/fe-core/src/main/java/org/apache/doris/analysis/ModifyTablePropertiesClause.java:
##########
@@ -142,6 +182,65 @@ public void analyze(Analyzer analyzer) throws AnalysisException {
|| properties.containsKey(PropertyAnalyzer.PROPERTIES_BINLOG_MAX_BYTES)
|| properties.containsKey(PropertyAnalyzer.PROPERTIES_BINLOG_MAX_HISTORY_NUMS)) {
// do nothing, will be alter in SchemaChangeHandler.updateBinlogConfig
+ } else if (properties.containsKey(PropertyAnalyzer.PROPERTIES_COMPACTION_POLICY)) {
+ String compactionPolicy = properties.getOrDefault(PropertyAnalyzer.PROPERTIES_COMPACTION_POLICY, "");
+ if (compactionPolicy != null
+ && !compactionPolicy.equals(PropertyAnalyzer.TIME_SERIES_COMPACTION_POLICY)
+ && !compactionPolicy.equals(PropertyAnalyzer.SIZE_BASED_COMPACTION_POLICY)) {
+ throw new AnalysisException(
+ "Table compaction policy only support for " + PropertyAnalyzer.TIME_SERIES_COMPACTION_POLICY
+ + " or " + PropertyAnalyzer.SIZE_BASED_COMPACTION_POLICY);
+ }
+ this.needTableStable = false;
+ setCompactionPolicy(compactionPolicy);
+ } else if (properties.containsKey(PropertyAnalyzer.PROPERTIES_TIME_SERIES_COMPACTION_GOAL_SIZE_MBYTES)) {
+ long goalSizeMbytes;
+ String goalSizeMbytesStr = properties
+ .get(PropertyAnalyzer.PROPERTIES_TIME_SERIES_COMPACTION_GOAL_SIZE_MBYTES);
+ try {
+ goalSizeMbytes = Long.parseLong(goalSizeMbytesStr);
+ if (goalSizeMbytes < 0) {
+ throw new AnalysisException("Invalid time_series_compaction_goal_size_mbytes format: "
+ + goalSizeMbytesStr);
+ }
+ } catch (NumberFormatException e) {
+ throw new AnalysisException("Invalid time_series_compaction_goal_size_mbytes format: "
+ + goalSizeMbytesStr);
+ }
+ this.needTableStable = false;
+ setTimeSeriesCompactionGoalSizeMbytes(goalSizeMbytes);
+ } else if (properties.containsKey(PropertyAnalyzer.PROPERTIES_TIME_SERIES_COMPACTION_FILE_COUNT_THRESHOLD)) {
+ long fileCountThreshold;
+ String fileCountThresholdStr = properties
+ .get(PropertyAnalyzer.PROPERTIES_TIME_SERIES_COMPACTION_FILE_COUNT_THRESHOLD);
+ try {
+ fileCountThreshold = Long.parseLong(fileCountThresholdStr);
+ if (fileCountThreshold < 0) {
+ throw new AnalysisException("Invalid time_series_compaction_file_count_threshold format: "
+ + fileCountThresholdStr);
+ }
+ } catch (NumberFormatException e) {
+ throw new AnalysisException("Invalid time_series_compaction_file_count_threshold format: "
+ + fileCountThresholdStr);
+ }
+ this.needTableStable = false;
+ setTimeSeriesCompactionFileCountThreshold(fileCountThreshold);
+ } else if (properties.containsKey(PropertyAnalyzer.PROPERTIES_TIME_SERIES_COMPACTION_TIME_THRESHOLD_SECONDS)) {
+ long timeThresholdSeconds;
+ String timeThresholdSecondsStr = properties
+ .get(PropertyAnalyzer.PROPERTIES_TIME_SERIES_COMPACTION_TIME_THRESHOLD_SECONDS);
+ try {
+ timeThresholdSeconds = Long.parseLong(timeThresholdSecondsStr);
+ if (timeThresholdSeconds < 0) {
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] xiaokang commented on pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
Posted by "xiaokang (via GitHub)" <gi...@apache.org>.
xiaokang commented on PR #22159:
URL: https://github.com/apache/doris/pull/22159#issuecomment-1660365312
trans to https://github.com/apache/doris/pull/22461
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #22159:
URL: https://github.com/apache/doris/pull/22159#issuecomment-1651256873
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.58 seconds
stream load tsv: 505 seconds loaded 74807831229 Bytes, about 141 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.6 seconds inserted 10000000 Rows, about 337K ops/s
storage size: 17166697669 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #22159:
URL: https://github.com/apache/doris/pull/22159#issuecomment-1652856904
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.88 seconds
stream load tsv: 508 seconds loaded 74807831229 Bytes, about 140 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.1 seconds inserted 10000000 Rows, about 343K ops/s
storage size: 17166264232 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #22159:
URL: https://github.com/apache/doris/pull/22159#issuecomment-1657124753
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] xiaokang commented on a diff in pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
Posted by "xiaokang (via GitHub)" <gi...@apache.org>.
xiaokang commented on code in PR #22159:
URL: https://github.com/apache/doris/pull/22159#discussion_r1278267673
##########
gensrc/proto/olap_file.proto:
##########
@@ -236,6 +236,10 @@ message TabletSchemaPB {
repeated string partial_update_input_columns = 21;
optional bool enable_single_replica_compaction = 22 [default=false];
optional bool skip_write_index_on_load = 23 [default=false];
+ optional string compaction_policy = 24;
Review Comment:
default value
##########
fe/fe-core/src/main/java/org/apache/doris/analysis/ModifyTablePropertiesClause.java:
##########
@@ -140,6 +180,49 @@ public void analyze(Analyzer analyzer) throws AnalysisException {
|| properties.containsKey(PropertyAnalyzer.PROPERTIES_BINLOG_MAX_BYTES)
|| properties.containsKey(PropertyAnalyzer.PROPERTIES_BINLOG_MAX_HISTORY_NUMS)) {
// do nothing, will be alter in SchemaChangeHandler.updateBinlogConfig
+ } else if (properties.containsKey(PropertyAnalyzer.PROPERTIES_COMPACTION_POLICY)) {
+ String compactionPolicy = properties.getOrDefault(PropertyAnalyzer.PROPERTIES_COMPACTION_POLICY, "");
+ if (!Strings.isNullOrEmpty(compactionPolicy)
+ && !compactionPolicy.equals("time_series") && !compactionPolicy.equals("size_based")) {
+ throw new AnalysisException(
+ "Table compaction policy only support for time_series or size_based");
+ }
+ this.needTableStable = false;
+ setCompactionPolicy(compactionPolicy);
+ } else if (properties.containsKey(PropertyAnalyzer.PROPERTIES_TIME_SERIES_COMPACTION_GOAL_SIZE_MBYTES)) {
+ long goalSizeMbytes;
+ String goalSizeMbytesStr = properties
+ .get(PropertyAnalyzer.PROPERTIES_TIME_SERIES_COMPACTION_GOAL_SIZE_MBYTES);
+ try {
+ goalSizeMbytes = Long.parseLong(goalSizeMbytesStr);
+ } catch (NumberFormatException e) {
+ throw new AnalysisException("Invalid time_series_compaction_goal_size_mbytes format: "
+ + goalSizeMbytesStr);
+ }
+ this.needTableStable = false;
+ setTimeSeriesCompactionGoalSizeMbytes(goalSizeMbytes);
+ } else if (properties.containsKey(PropertyAnalyzer.PROPERTIES_TIME_SERIES_COMPACTION_FILE_COUNT_THRESHOLD)) {
+ long fileCountThreshold;
+ String fileCountThresholdStr = properties
+ .get(PropertyAnalyzer.PROPERTIES_TIME_SERIES_COMPACTION_FILE_COUNT_THRESHOLD);
+ try {
+ fileCountThreshold = Long.parseLong(fileCountThresholdStr);
+ } catch (NumberFormatException e) {
+ throw new AnalysisException("Invalid time_series_compaction_file_count_threshold format: "
+ + fileCountThresholdStr);
+ }
+ this.needTableStable = false;
+ setTimeSeriesCompactionFileCountThreshold(fileCountThreshold);
+ } else if (properties.containsKey(PropertyAnalyzer.PROPERTIES_TIME_SERIES_COMPACTION_TIME_THRESHOLD_SECONDS)) {
+ long timeThresholdSeconds;
+ try {
+ timeThresholdSeconds = Long.parseLong(properties
Review Comment:
check > 0
##########
be/src/olap/compaction.cpp:
##########
@@ -148,8 +148,10 @@ int64_t Compaction::get_avg_segment_rows() {
// input_rowsets_size is total disk_size of input_rowset, this size is the
// final size after codec and compress, so expect dest segment file size
// in disk is config::vertical_compaction_max_segment_size
- if (config::compaction_policy == CUMULATIVE_TIME_SERIES_POLICY) {
- return (config::time_series_compaction_goal_size_mbytes * 1024 * 1024 * 2) /
+ if (_tablet->tablet_meta()->tablet_schema()->compaction_policy() ==
Review Comment:
use a variable for tablet->tablet_meta()->tablet_schema() to simplify code
##########
be/src/olap/cumulative_compaction_policy.cpp:
##########
@@ -368,11 +368,13 @@ int64_t SizeBasedCumulativeCompactionPolicy::_level_size(const int64_t size) {
}
std::shared_ptr<CumulativeCompactionPolicy>
-CumulativeCompactionPolicyFactory::create_cumulative_compaction_policy() {
Review Comment:
add an argument policy
##########
fe/fe-core/src/main/java/org/apache/doris/catalog/TableProperty.java:
##########
@@ -86,6 +86,14 @@ public class TableProperty implements Writable {
private boolean skipWriteIndexOnLoad = false;
+ private String compactionPolicy = "";
Review Comment:
default value
##########
fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java:
##########
@@ -3120,6 +3120,36 @@ public static void getDdlStmt(DdlStmt ddlStmt, String dbName, TableIf table, Lis
sb.append(olapTable.skipWriteIndexOnLoad()).append("\"");
}
+ // compaction policy
+ if (olapTable.getCompactionPolicy() != null && !olapTable.getCompactionPolicy().equals("")) {
+ sb.append(",\n\"").append(PropertyAnalyzer.PROPERTIES_COMPACTION_POLICY).append("\" = \"");
+ sb.append(olapTable.getCompactionPolicy()).append("\"");
+ }
+
+ // time series compaction goal size
+ if (olapTable.getTimeSeriesCompactionGoalSizeMbytes() != null
+ && olapTable.getTimeSeriesCompactionGoalSizeMbytes() != 512) {
Review Comment:
use const variable instead of literal
##########
fe/fe-core/src/main/java/org/apache/doris/analysis/ModifyTablePropertiesClause.java:
##########
@@ -140,6 +180,49 @@ public void analyze(Analyzer analyzer) throws AnalysisException {
|| properties.containsKey(PropertyAnalyzer.PROPERTIES_BINLOG_MAX_BYTES)
|| properties.containsKey(PropertyAnalyzer.PROPERTIES_BINLOG_MAX_HISTORY_NUMS)) {
// do nothing, will be alter in SchemaChangeHandler.updateBinlogConfig
+ } else if (properties.containsKey(PropertyAnalyzer.PROPERTIES_COMPACTION_POLICY)) {
+ String compactionPolicy = properties.getOrDefault(PropertyAnalyzer.PROPERTIES_COMPACTION_POLICY, "");
+ if (!Strings.isNullOrEmpty(compactionPolicy)
+ && !compactionPolicy.equals("time_series") && !compactionPolicy.equals("size_based")) {
Review Comment:
use const variable instead of literal
##########
fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java:
##########
@@ -3120,6 +3120,36 @@ public static void getDdlStmt(DdlStmt ddlStmt, String dbName, TableIf table, Lis
sb.append(olapTable.skipWriteIndexOnLoad()).append("\"");
}
+ // compaction policy
+ if (olapTable.getCompactionPolicy() != null && !olapTable.getCompactionPolicy().equals("")) {
+ sb.append(",\n\"").append(PropertyAnalyzer.PROPERTIES_COMPACTION_POLICY).append("\" = \"");
+ sb.append(olapTable.getCompactionPolicy()).append("\"");
+ }
+
+ // time series compaction goal size
+ if (olapTable.getTimeSeriesCompactionGoalSizeMbytes() != null
+ && olapTable.getTimeSeriesCompactionGoalSizeMbytes() != 512) {
+ sb.append(",\n\"").append(PropertyAnalyzer
+ .PROPERTIES_TIME_SERIES_COMPACTION_GOAL_SIZE_MBYTES).append("\" = \"");
+ sb.append(olapTable.getTimeSeriesCompactionGoalSizeMbytes()).append("\"");
+ }
+
+ // time series compaction file count threshold
+ if (olapTable.getTimeSeriesCompactionFileCountThreshold() != null
+ && olapTable.getTimeSeriesCompactionFileCountThreshold() != 2000) {
Review Comment:
use const variable instead of literal
##########
fe/fe-core/src/main/java/org/apache/doris/common/util/PropertyAnalyzer.java:
##########
@@ -595,6 +605,82 @@ public static Boolean analyzeSkipWriteIndexOnLoad(Map<String, String> properties
+ " must be `true` or `false`");
}
+ public static String analyzeCompactionPolicy(Map<String, String> properties) throws AnalysisException {
+ if (properties == null || properties.isEmpty()) {
+ return "";
+ }
+ String compactionPolicy = "";
Review Comment:
default value
##########
fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java:
##########
@@ -3120,6 +3120,36 @@ public static void getDdlStmt(DdlStmt ddlStmt, String dbName, TableIf table, Lis
sb.append(olapTable.skipWriteIndexOnLoad()).append("\"");
}
+ // compaction policy
+ if (olapTable.getCompactionPolicy() != null && !olapTable.getCompactionPolicy().equals("")) {
+ sb.append(",\n\"").append(PropertyAnalyzer.PROPERTIES_COMPACTION_POLICY).append("\" = \"");
+ sb.append(olapTable.getCompactionPolicy()).append("\"");
+ }
+
+ // time series compaction goal size
+ if (olapTable.getTimeSeriesCompactionGoalSizeMbytes() != null
+ && olapTable.getTimeSeriesCompactionGoalSizeMbytes() != 512) {
+ sb.append(",\n\"").append(PropertyAnalyzer
+ .PROPERTIES_TIME_SERIES_COMPACTION_GOAL_SIZE_MBYTES).append("\" = \"");
+ sb.append(olapTable.getTimeSeriesCompactionGoalSizeMbytes()).append("\"");
+ }
+
+ // time series compaction file count threshold
+ if (olapTable.getTimeSeriesCompactionFileCountThreshold() != null
+ && olapTable.getTimeSeriesCompactionFileCountThreshold() != 2000) {
+ sb.append(",\n\"").append(PropertyAnalyzer
+ .PROPERTIES_TIME_SERIES_COMPACTION_FILE_COUNT_THRESHOLD).append("\" = \"");
+ sb.append(olapTable.getTimeSeriesCompactionFileCountThreshold()).append("\"");
+ }
+
+ // time series compaction time threshold
+ if (olapTable.getTimeSeriesCompactionTimeThresholdSeconds() != null
+ && olapTable.getTimeSeriesCompactionTimeThresholdSeconds() != 3600) {
Review Comment:
use const variable instead of literal
##########
fe/fe-core/src/main/java/org/apache/doris/alter/SchemaChangeHandler.java:
##########
@@ -2137,13 +2137,40 @@ public void updateTableProperties(Database db, String tableName, Map<String, Str
}
long storagePolicyId = storagePolicyNameToId(storagePolicy);
- if (isInMemory < 0 && storagePolicyId < 0) {
- LOG.info("Properties already up-to-date");
- return;
+ // if (isInMemory < 0 && storagePolicyId < 0) {
Review Comment:
why delete the code
##########
fe/fe-core/src/main/java/org/apache/doris/analysis/ModifyTablePropertiesClause.java:
##########
@@ -140,6 +180,49 @@ public void analyze(Analyzer analyzer) throws AnalysisException {
|| properties.containsKey(PropertyAnalyzer.PROPERTIES_BINLOG_MAX_BYTES)
|| properties.containsKey(PropertyAnalyzer.PROPERTIES_BINLOG_MAX_HISTORY_NUMS)) {
// do nothing, will be alter in SchemaChangeHandler.updateBinlogConfig
+ } else if (properties.containsKey(PropertyAnalyzer.PROPERTIES_COMPACTION_POLICY)) {
+ String compactionPolicy = properties.getOrDefault(PropertyAnalyzer.PROPERTIES_COMPACTION_POLICY, "");
+ if (!Strings.isNullOrEmpty(compactionPolicy)
+ && !compactionPolicy.equals("time_series") && !compactionPolicy.equals("size_based")) {
+ throw new AnalysisException(
+ "Table compaction policy only support for time_series or size_based");
+ }
+ this.needTableStable = false;
+ setCompactionPolicy(compactionPolicy);
+ } else if (properties.containsKey(PropertyAnalyzer.PROPERTIES_TIME_SERIES_COMPACTION_GOAL_SIZE_MBYTES)) {
+ long goalSizeMbytes;
+ String goalSizeMbytesStr = properties
+ .get(PropertyAnalyzer.PROPERTIES_TIME_SERIES_COMPACTION_GOAL_SIZE_MBYTES);
+ try {
+ goalSizeMbytes = Long.parseLong(goalSizeMbytesStr);
Review Comment:
check size > 0
##########
fe/fe-core/src/main/java/org/apache/doris/analysis/ModifyTablePropertiesClause.java:
##########
@@ -140,6 +180,49 @@ public void analyze(Analyzer analyzer) throws AnalysisException {
|| properties.containsKey(PropertyAnalyzer.PROPERTIES_BINLOG_MAX_BYTES)
|| properties.containsKey(PropertyAnalyzer.PROPERTIES_BINLOG_MAX_HISTORY_NUMS)) {
// do nothing, will be alter in SchemaChangeHandler.updateBinlogConfig
+ } else if (properties.containsKey(PropertyAnalyzer.PROPERTIES_COMPACTION_POLICY)) {
+ String compactionPolicy = properties.getOrDefault(PropertyAnalyzer.PROPERTIES_COMPACTION_POLICY, "");
+ if (!Strings.isNullOrEmpty(compactionPolicy)
+ && !compactionPolicy.equals("time_series") && !compactionPolicy.equals("size_based")) {
+ throw new AnalysisException(
+ "Table compaction policy only support for time_series or size_based");
+ }
+ this.needTableStable = false;
+ setCompactionPolicy(compactionPolicy);
+ } else if (properties.containsKey(PropertyAnalyzer.PROPERTIES_TIME_SERIES_COMPACTION_GOAL_SIZE_MBYTES)) {
+ long goalSizeMbytes;
+ String goalSizeMbytesStr = properties
+ .get(PropertyAnalyzer.PROPERTIES_TIME_SERIES_COMPACTION_GOAL_SIZE_MBYTES);
+ try {
+ goalSizeMbytes = Long.parseLong(goalSizeMbytesStr);
+ } catch (NumberFormatException e) {
+ throw new AnalysisException("Invalid time_series_compaction_goal_size_mbytes format: "
+ + goalSizeMbytesStr);
+ }
+ this.needTableStable = false;
+ setTimeSeriesCompactionGoalSizeMbytes(goalSizeMbytes);
+ } else if (properties.containsKey(PropertyAnalyzer.PROPERTIES_TIME_SERIES_COMPACTION_FILE_COUNT_THRESHOLD)) {
+ long fileCountThreshold;
+ String fileCountThresholdStr = properties
+ .get(PropertyAnalyzer.PROPERTIES_TIME_SERIES_COMPACTION_FILE_COUNT_THRESHOLD);
+ try {
+ fileCountThreshold = Long.parseLong(fileCountThresholdStr);
Review Comment:
check > 0
##########
be/src/olap/cumulative_compaction_time_series_policy.cpp:
##########
@@ -63,13 +63,16 @@ uint32_t TimeSeriesCumulativeCompactionPolicy::calc_cumulative_compaction_score(
return 0;
}
- // Condition 1: the size of input files for compaction meets the requirement of parameter _compaction_goal_size
- if (total_size >= (config::time_series_compaction_goal_size_mbytes * 1024 * 1024)) {
+ // Condition 1: the size of input files for compaction meets the requirement of parameter compaction_goal_size
+ if (total_size >=
+ (tablet->tablet_meta()->tablet_schema()->time_series_compaction_goal_size_mbytes() * 1024 *
Review Comment:
use a local variable to store tablet->tablet_meta()->tablet_schema() to simplify code
##########
be/src/olap/cumulative_compaction_policy.cpp:
##########
@@ -368,11 +368,13 @@ int64_t SizeBasedCumulativeCompactionPolicy::_level_size(const int64_t size) {
}
std::shared_ptr<CumulativeCompactionPolicy>
-CumulativeCompactionPolicyFactory::create_cumulative_compaction_policy() {
Review Comment:
just keep single create_cumulative_compaction_policy() instead of two create functions
##########
be/src/agent/task_worker_pool.cpp:
##########
@@ -433,6 +433,63 @@ void TaskWorkerPool::_update_tablet_meta_worker_thread_callback() {
tablet->tablet_schema_unlocked()->set_is_in_memory(tablet_meta_info.is_in_memory);
need_to_save = true;
}
+ if (tablet_meta_info.__isset.compaction_policy) {
+ tablet->tablet_meta()->mutable_tablet_schema()->set_compaction_policy(
+ tablet_meta_info.compaction_policy);
+ std::shared_lock rlock(tablet->get_header_lock());
Review Comment:
lock problem:
1. why not write lock
2. why not lock before mutable_tablet_schema()->set_compaction_policy
##########
be/src/agent/task_worker_pool.cpp:
##########
@@ -433,6 +433,63 @@ void TaskWorkerPool::_update_tablet_meta_worker_thread_callback() {
tablet->tablet_schema_unlocked()->set_is_in_memory(tablet_meta_info.is_in_memory);
need_to_save = true;
}
+ if (tablet_meta_info.__isset.compaction_policy) {
+ tablet->tablet_meta()->mutable_tablet_schema()->set_compaction_policy(
+ tablet_meta_info.compaction_policy);
+ std::shared_lock rlock(tablet->get_header_lock());
+ for (auto& rowset_meta : tablet->tablet_meta()->all_mutable_rs_metas()) {
+ rowset_meta->tablet_schema()->set_compaction_policy(
+ tablet_meta_info.compaction_policy);
+ }
+ tablet->tablet_schema_unlocked()->set_compaction_policy(
Review Comment:
what's difference to mutable_tablet_schema()->set_compaction_policy?
##########
be/src/olap/olap_server.cpp:
##########
@@ -876,9 +877,12 @@ std::vector<TabletSharedPtr> StorageEngine::_generate_compaction_tasks(
}
void StorageEngine::_update_cumulative_compaction_policy() {
- if (_cumulative_compaction_policy == nullptr) {
- _cumulative_compaction_policy =
- CumulativeCompactionPolicyFactory::create_cumulative_compaction_policy();
+ if (_all_cumulative_compaction_policy.size() < 2) {
Review Comment:
check size 2 is tricky. check _all_cumulative_compaction_policy.count(policyxx).
BTW, name _cumulative_compaction_policies is better
##########
be/src/olap/olap_server.cpp:
##########
@@ -989,7 +993,14 @@ Status StorageEngine::submit_compaction_task(TabletSharedPtr tablet, CompactionT
bool force) {
_update_cumulative_compaction_policy();
if (tablet->get_cumulative_compaction_policy() == nullptr) {
- tablet->set_cumulative_compaction_policy(_cumulative_compaction_policy);
Review Comment:
why change to complex logic
##########
be/src/olap/storage_engine.h:
##########
@@ -462,7 +462,9 @@ class StorageEngine {
std::shared_ptr<StreamLoadRecorder> _stream_load_recorder;
- std::shared_ptr<CumulativeCompactionPolicy> _cumulative_compaction_policy;
+ // we use unordered_map to store all cumulative compaction policy sharded ptr
Review Comment:
why?
##########
be/src/http/action/compaction_action.cpp:
##########
@@ -198,7 +199,12 @@ Status CompactionAction::_execute_compaction_callback(TabletSharedPtr tablet,
timer.start();
std::shared_ptr<CumulativeCompactionPolicy> cumulative_compaction_policy =
Review Comment:
encapsulate the if else logic to create_cumulative_compaction_policy() function
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #22159:
URL: https://github.com/apache/doris/pull/22159#issuecomment-1657057905
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.02 seconds
stream load tsv: 503 seconds loaded 74807831229 Bytes, about 141 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.4 seconds inserted 10000000 Rows, about 340K ops/s
storage size: 17165270264 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #22159:
URL: https://github.com/apache/doris/pull/22159#issuecomment-1657172563
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] csun5285 commented on pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
Posted by "csun5285 (via GitHub)" <gi...@apache.org>.
csun5285 commented on PR #22159:
URL: https://github.com/apache/doris/pull/22159#issuecomment-1657065167
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #22159:
URL: https://github.com/apache/doris/pull/22159#issuecomment-1657066170
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] csun5285 commented on pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
Posted by "csun5285 (via GitHub)" <gi...@apache.org>.
csun5285 commented on PR #22159:
URL: https://github.com/apache/doris/pull/22159#issuecomment-1657213691
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #22159:
URL: https://github.com/apache/doris/pull/22159#issuecomment-1658252988
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #22159:
URL: https://github.com/apache/doris/pull/22159#issuecomment-1657072605
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 48.66 seconds
stream load tsv: 506 seconds loaded 74807831229 Bytes, about 140 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 34 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 36.5 seconds inserted 10000000 Rows, about 273K ops/s
storage size: 17167360166 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] csun5285 commented on a diff in pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
Posted by "csun5285 (via GitHub)" <gi...@apache.org>.
csun5285 commented on code in PR #22159:
URL: https://github.com/apache/doris/pull/22159#discussion_r1279005566
##########
gensrc/thrift/AgentService.thrift:
##########
@@ -142,6 +142,10 @@ struct TCreateTabletReq {
19: optional bool enable_unique_key_merge_on_write = false
20: optional i64 storage_policy_id
21: optional TBinlogConfig binlog_config
+ 22: optional string compaction_policy = "size_based"
+ 23: optional i64 time_series_compaction_goal_size_mbytes = 512
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] csun5285 commented on pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
Posted by "csun5285 (via GitHub)" <gi...@apache.org>.
csun5285 commented on PR #22159:
URL: https://github.com/apache/doris/pull/22159#issuecomment-1652791455
run p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #22159:
URL: https://github.com/apache/doris/pull/22159#issuecomment-1651046056
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] csun5285 commented on a diff in pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
Posted by "csun5285 (via GitHub)" <gi...@apache.org>.
csun5285 commented on code in PR #22159:
URL: https://github.com/apache/doris/pull/22159#discussion_r1278307898
##########
be/src/http/action/compaction_action.cpp:
##########
@@ -198,7 +199,12 @@ Status CompactionAction::_execute_compaction_callback(TabletSharedPtr tablet,
timer.start();
std::shared_ptr<CumulativeCompactionPolicy> cumulative_compaction_policy =
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] csun5285 commented on a diff in pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
Posted by "csun5285 (via GitHub)" <gi...@apache.org>.
csun5285 commented on code in PR #22159:
URL: https://github.com/apache/doris/pull/22159#discussion_r1278509531
##########
fe/fe-core/src/main/java/org/apache/doris/common/util/PropertyAnalyzer.java:
##########
@@ -595,6 +605,82 @@ public static Boolean analyzeSkipWriteIndexOnLoad(Map<String, String> properties
+ " must be `true` or `false`");
}
+ public static String analyzeCompactionPolicy(Map<String, String> properties) throws AnalysisException {
+ if (properties == null || properties.isEmpty()) {
+ return "";
+ }
+ String compactionPolicy = "";
Review Comment:
done
##########
gensrc/proto/olap_file.proto:
##########
@@ -236,6 +236,10 @@ message TabletSchemaPB {
repeated string partial_update_input_columns = 21;
optional bool enable_single_replica_compaction = 22 [default=false];
optional bool skip_write_index_on_load = 23 [default=false];
+ optional string compaction_policy = 24;
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #22159:
URL: https://github.com/apache/doris/pull/22159#issuecomment-1657172558
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] csun5285 commented on pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
Posted by "csun5285 (via GitHub)" <gi...@apache.org>.
csun5285 commented on PR #22159:
URL: https://github.com/apache/doris/pull/22159#issuecomment-1658060113
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #22159:
URL: https://github.com/apache/doris/pull/22159#issuecomment-1658138724
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.02 seconds
stream load tsv: 506 seconds loaded 74807831229 Bytes, about 140 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.5 seconds inserted 10000000 Rows, about 338K ops/s
storage size: 17165776262 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] csun5285 commented on pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
Posted by "csun5285 (via GitHub)" <gi...@apache.org>.
csun5285 commented on PR #22159:
URL: https://github.com/apache/doris/pull/22159#issuecomment-1658127074
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #22159:
URL: https://github.com/apache/doris/pull/22159#issuecomment-1659581826
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] csun5285 commented on a diff in pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
Posted by "csun5285 (via GitHub)" <gi...@apache.org>.
csun5285 commented on code in PR #22159:
URL: https://github.com/apache/doris/pull/22159#discussion_r1279017706
##########
be/src/olap/tablet_manager.cpp:
##########
@@ -729,7 +731,10 @@ TabletSharedPtr TabletManager::find_best_tablet_to_compaction(
continue;
}
}
-
+ auto cumulative_compaction_policy =
+ tablet_ptr->tablet_meta()->compaction_policy() == CUMULATIVE_TIME_SERIES_POLICY
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] csun5285 commented on pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
Posted by "csun5285 (via GitHub)" <gi...@apache.org>.
csun5285 commented on PR #22159:
URL: https://github.com/apache/doris/pull/22159#issuecomment-1658066587
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] csun5285 commented on pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
Posted by "csun5285 (via GitHub)" <gi...@apache.org>.
csun5285 commented on PR #22159:
URL: https://github.com/apache/doris/pull/22159#issuecomment-1658101553
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #22159:
URL: https://github.com/apache/doris/pull/22159#issuecomment-1658109988
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #22159:
URL: https://github.com/apache/doris/pull/22159#issuecomment-1657181347
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.5 seconds
stream load tsv: 502 seconds loaded 74807831229 Bytes, about 142 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.2 seconds inserted 10000000 Rows, about 342K ops/s
storage size: 17169210993 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #22159:
URL: https://github.com/apache/doris/pull/22159#issuecomment-1657214959
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] csun5285 commented on a diff in pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
Posted by "csun5285 (via GitHub)" <gi...@apache.org>.
csun5285 commented on code in PR #22159:
URL: https://github.com/apache/doris/pull/22159#discussion_r1278307752
##########
fe/fe-core/src/main/java/org/apache/doris/analysis/ModifyTablePropertiesClause.java:
##########
@@ -140,6 +180,49 @@ public void analyze(Analyzer analyzer) throws AnalysisException {
|| properties.containsKey(PropertyAnalyzer.PROPERTIES_BINLOG_MAX_BYTES)
|| properties.containsKey(PropertyAnalyzer.PROPERTIES_BINLOG_MAX_HISTORY_NUMS)) {
// do nothing, will be alter in SchemaChangeHandler.updateBinlogConfig
+ } else if (properties.containsKey(PropertyAnalyzer.PROPERTIES_COMPACTION_POLICY)) {
+ String compactionPolicy = properties.getOrDefault(PropertyAnalyzer.PROPERTIES_COMPACTION_POLICY, "");
+ if (!Strings.isNullOrEmpty(compactionPolicy)
+ && !compactionPolicy.equals("time_series") && !compactionPolicy.equals("size_based")) {
Review Comment:
done
##########
fe/fe-core/src/main/java/org/apache/doris/analysis/ModifyTablePropertiesClause.java:
##########
@@ -140,6 +180,49 @@ public void analyze(Analyzer analyzer) throws AnalysisException {
|| properties.containsKey(PropertyAnalyzer.PROPERTIES_BINLOG_MAX_BYTES)
|| properties.containsKey(PropertyAnalyzer.PROPERTIES_BINLOG_MAX_HISTORY_NUMS)) {
// do nothing, will be alter in SchemaChangeHandler.updateBinlogConfig
+ } else if (properties.containsKey(PropertyAnalyzer.PROPERTIES_COMPACTION_POLICY)) {
+ String compactionPolicy = properties.getOrDefault(PropertyAnalyzer.PROPERTIES_COMPACTION_POLICY, "");
+ if (!Strings.isNullOrEmpty(compactionPolicy)
+ && !compactionPolicy.equals("time_series") && !compactionPolicy.equals("size_based")) {
+ throw new AnalysisException(
+ "Table compaction policy only support for time_series or size_based");
+ }
+ this.needTableStable = false;
+ setCompactionPolicy(compactionPolicy);
+ } else if (properties.containsKey(PropertyAnalyzer.PROPERTIES_TIME_SERIES_COMPACTION_GOAL_SIZE_MBYTES)) {
+ long goalSizeMbytes;
+ String goalSizeMbytesStr = properties
+ .get(PropertyAnalyzer.PROPERTIES_TIME_SERIES_COMPACTION_GOAL_SIZE_MBYTES);
+ try {
+ goalSizeMbytes = Long.parseLong(goalSizeMbytesStr);
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] csun5285 commented on pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
Posted by "csun5285 (via GitHub)" <gi...@apache.org>.
csun5285 commented on PR #22159:
URL: https://github.com/apache/doris/pull/22159#issuecomment-1651622775
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #22159:
URL: https://github.com/apache/doris/pull/22159#issuecomment-1649889622
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.25 seconds
stream load tsv: 507 seconds loaded 74807831229 Bytes, about 140 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.1 seconds inserted 10000000 Rows, about 343K ops/s
storage size: 17168956458 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] csun5285 commented on a diff in pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
Posted by "csun5285 (via GitHub)" <gi...@apache.org>.
csun5285 commented on code in PR #22159:
URL: https://github.com/apache/doris/pull/22159#discussion_r1278307723
##########
fe/fe-core/src/main/java/org/apache/doris/catalog/TableProperty.java:
##########
@@ -86,6 +86,14 @@ public class TableProperty implements Writable {
private boolean skipWriteIndexOnLoad = false;
+ private String compactionPolicy = "";
Review Comment:
done
##########
fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java:
##########
@@ -3120,6 +3120,36 @@ public static void getDdlStmt(DdlStmt ddlStmt, String dbName, TableIf table, Lis
sb.append(olapTable.skipWriteIndexOnLoad()).append("\"");
}
+ // compaction policy
+ if (olapTable.getCompactionPolicy() != null && !olapTable.getCompactionPolicy().equals("")) {
+ sb.append(",\n\"").append(PropertyAnalyzer.PROPERTIES_COMPACTION_POLICY).append("\" = \"");
+ sb.append(olapTable.getCompactionPolicy()).append("\"");
+ }
+
+ // time series compaction goal size
+ if (olapTable.getTimeSeriesCompactionGoalSizeMbytes() != null
+ && olapTable.getTimeSeriesCompactionGoalSizeMbytes() != 512) {
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #22159:
URL: https://github.com/apache/doris/pull/22159#issuecomment-1657045195
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #22159:
URL: https://github.com/apache/doris/pull/22159#issuecomment-1657098983
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] csun5285 commented on pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
Posted by "csun5285 (via GitHub)" <gi...@apache.org>.
csun5285 commented on PR #22159:
URL: https://github.com/apache/doris/pull/22159#issuecomment-1657123314
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #22159:
URL: https://github.com/apache/doris/pull/22159#issuecomment-1658075593
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #22159:
URL: https://github.com/apache/doris/pull/22159#issuecomment-1658140814
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #22159:
URL: https://github.com/apache/doris/pull/22159#issuecomment-1658366073
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] xiaokang closed pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
Posted by "xiaokang (via GitHub)" <gi...@apache.org>.
xiaokang closed pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
URL: https://github.com/apache/doris/pull/22159
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #22159:
URL: https://github.com/apache/doris/pull/22159#issuecomment-1659651039
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.82 seconds
stream load tsv: 511 seconds loaded 74807831229 Bytes, about 139 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 30 seconds loaded 861443392 Bytes, about 27 MB/s
insert into select: 29.2 seconds inserted 10000000 Rows, about 342K ops/s
storage size: 17162297370 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #22159:
URL: https://github.com/apache/doris/pull/22159#issuecomment-1647633387
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] csun5285 commented on pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
Posted by "csun5285 (via GitHub)" <gi...@apache.org>.
csun5285 commented on PR #22159:
URL: https://github.com/apache/doris/pull/22159#issuecomment-1649755305
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #22159:
URL: https://github.com/apache/doris/pull/22159#issuecomment-1657131916
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.74 seconds
stream load tsv: 504 seconds loaded 74807831229 Bytes, about 141 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 30 seconds loaded 861443392 Bytes, about 27 MB/s
insert into select: 29.1 seconds inserted 10000000 Rows, about 343K ops/s
storage size: 17164983479 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] csun5285 commented on a diff in pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
Posted by "csun5285 (via GitHub)" <gi...@apache.org>.
csun5285 commented on code in PR #22159:
URL: https://github.com/apache/doris/pull/22159#discussion_r1278309204
##########
fe/fe-core/src/main/java/org/apache/doris/alter/SchemaChangeHandler.java:
##########
@@ -2137,13 +2137,40 @@ public void updateTableProperties(Database db, String tableName, Map<String, Str
}
long storagePolicyId = storagePolicyNameToId(storagePolicy);
- if (isInMemory < 0 && storagePolicyId < 0) {
- LOG.info("Properties already up-to-date");
- return;
+ // if (isInMemory < 0 && storagePolicyId < 0) {
Review Comment:
If the two properties (in_memory and storage_policy) do not change, there might not execute a schema change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] xiaokang commented on pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
Posted by "xiaokang (via GitHub)" <gi...@apache.org>.
xiaokang commented on PR #22159:
URL: https://github.com/apache/doris/pull/22159#issuecomment-1656667170
regression test need to be added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] csun5285 commented on a diff in pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
Posted by "csun5285 (via GitHub)" <gi...@apache.org>.
csun5285 commented on code in PR #22159:
URL: https://github.com/apache/doris/pull/22159#discussion_r1278307762
##########
fe/fe-core/src/main/java/org/apache/doris/analysis/ModifyTablePropertiesClause.java:
##########
@@ -140,6 +180,49 @@ public void analyze(Analyzer analyzer) throws AnalysisException {
|| properties.containsKey(PropertyAnalyzer.PROPERTIES_BINLOG_MAX_BYTES)
|| properties.containsKey(PropertyAnalyzer.PROPERTIES_BINLOG_MAX_HISTORY_NUMS)) {
// do nothing, will be alter in SchemaChangeHandler.updateBinlogConfig
+ } else if (properties.containsKey(PropertyAnalyzer.PROPERTIES_COMPACTION_POLICY)) {
+ String compactionPolicy = properties.getOrDefault(PropertyAnalyzer.PROPERTIES_COMPACTION_POLICY, "");
+ if (!Strings.isNullOrEmpty(compactionPolicy)
+ && !compactionPolicy.equals("time_series") && !compactionPolicy.equals("size_based")) {
+ throw new AnalysisException(
+ "Table compaction policy only support for time_series or size_based");
+ }
+ this.needTableStable = false;
+ setCompactionPolicy(compactionPolicy);
+ } else if (properties.containsKey(PropertyAnalyzer.PROPERTIES_TIME_SERIES_COMPACTION_GOAL_SIZE_MBYTES)) {
+ long goalSizeMbytes;
+ String goalSizeMbytesStr = properties
+ .get(PropertyAnalyzer.PROPERTIES_TIME_SERIES_COMPACTION_GOAL_SIZE_MBYTES);
+ try {
+ goalSizeMbytes = Long.parseLong(goalSizeMbytesStr);
+ } catch (NumberFormatException e) {
+ throw new AnalysisException("Invalid time_series_compaction_goal_size_mbytes format: "
+ + goalSizeMbytesStr);
+ }
+ this.needTableStable = false;
+ setTimeSeriesCompactionGoalSizeMbytes(goalSizeMbytes);
+ } else if (properties.containsKey(PropertyAnalyzer.PROPERTIES_TIME_SERIES_COMPACTION_FILE_COUNT_THRESHOLD)) {
+ long fileCountThreshold;
+ String fileCountThresholdStr = properties
+ .get(PropertyAnalyzer.PROPERTIES_TIME_SERIES_COMPACTION_FILE_COUNT_THRESHOLD);
+ try {
+ fileCountThreshold = Long.parseLong(fileCountThresholdStr);
Review Comment:
done
##########
fe/fe-core/src/main/java/org/apache/doris/analysis/ModifyTablePropertiesClause.java:
##########
@@ -140,6 +180,49 @@ public void analyze(Analyzer analyzer) throws AnalysisException {
|| properties.containsKey(PropertyAnalyzer.PROPERTIES_BINLOG_MAX_BYTES)
|| properties.containsKey(PropertyAnalyzer.PROPERTIES_BINLOG_MAX_HISTORY_NUMS)) {
// do nothing, will be alter in SchemaChangeHandler.updateBinlogConfig
+ } else if (properties.containsKey(PropertyAnalyzer.PROPERTIES_COMPACTION_POLICY)) {
+ String compactionPolicy = properties.getOrDefault(PropertyAnalyzer.PROPERTIES_COMPACTION_POLICY, "");
+ if (!Strings.isNullOrEmpty(compactionPolicy)
+ && !compactionPolicy.equals("time_series") && !compactionPolicy.equals("size_based")) {
+ throw new AnalysisException(
+ "Table compaction policy only support for time_series or size_based");
+ }
+ this.needTableStable = false;
+ setCompactionPolicy(compactionPolicy);
+ } else if (properties.containsKey(PropertyAnalyzer.PROPERTIES_TIME_SERIES_COMPACTION_GOAL_SIZE_MBYTES)) {
+ long goalSizeMbytes;
+ String goalSizeMbytesStr = properties
+ .get(PropertyAnalyzer.PROPERTIES_TIME_SERIES_COMPACTION_GOAL_SIZE_MBYTES);
+ try {
+ goalSizeMbytes = Long.parseLong(goalSizeMbytesStr);
+ } catch (NumberFormatException e) {
+ throw new AnalysisException("Invalid time_series_compaction_goal_size_mbytes format: "
+ + goalSizeMbytesStr);
+ }
+ this.needTableStable = false;
+ setTimeSeriesCompactionGoalSizeMbytes(goalSizeMbytes);
+ } else if (properties.containsKey(PropertyAnalyzer.PROPERTIES_TIME_SERIES_COMPACTION_FILE_COUNT_THRESHOLD)) {
+ long fileCountThreshold;
+ String fileCountThresholdStr = properties
+ .get(PropertyAnalyzer.PROPERTIES_TIME_SERIES_COMPACTION_FILE_COUNT_THRESHOLD);
+ try {
+ fileCountThreshold = Long.parseLong(fileCountThresholdStr);
+ } catch (NumberFormatException e) {
+ throw new AnalysisException("Invalid time_series_compaction_file_count_threshold format: "
+ + fileCountThresholdStr);
+ }
+ this.needTableStable = false;
+ setTimeSeriesCompactionFileCountThreshold(fileCountThreshold);
+ } else if (properties.containsKey(PropertyAnalyzer.PROPERTIES_TIME_SERIES_COMPACTION_TIME_THRESHOLD_SECONDS)) {
+ long timeThresholdSeconds;
+ try {
+ timeThresholdSeconds = Long.parseLong(properties
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] csun5285 commented on a diff in pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
Posted by "csun5285 (via GitHub)" <gi...@apache.org>.
csun5285 commented on code in PR #22159:
URL: https://github.com/apache/doris/pull/22159#discussion_r1278307733
##########
fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java:
##########
@@ -3120,6 +3120,36 @@ public static void getDdlStmt(DdlStmt ddlStmt, String dbName, TableIf table, Lis
sb.append(olapTable.skipWriteIndexOnLoad()).append("\"");
}
+ // compaction policy
+ if (olapTable.getCompactionPolicy() != null && !olapTable.getCompactionPolicy().equals("")) {
+ sb.append(",\n\"").append(PropertyAnalyzer.PROPERTIES_COMPACTION_POLICY).append("\" = \"");
+ sb.append(olapTable.getCompactionPolicy()).append("\"");
+ }
+
+ // time series compaction goal size
+ if (olapTable.getTimeSeriesCompactionGoalSizeMbytes() != null
+ && olapTable.getTimeSeriesCompactionGoalSizeMbytes() != 512) {
+ sb.append(",\n\"").append(PropertyAnalyzer
+ .PROPERTIES_TIME_SERIES_COMPACTION_GOAL_SIZE_MBYTES).append("\" = \"");
+ sb.append(olapTable.getTimeSeriesCompactionGoalSizeMbytes()).append("\"");
+ }
+
+ // time series compaction file count threshold
+ if (olapTable.getTimeSeriesCompactionFileCountThreshold() != null
+ && olapTable.getTimeSeriesCompactionFileCountThreshold() != 2000) {
Review Comment:
done
##########
fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java:
##########
@@ -3120,6 +3120,36 @@ public static void getDdlStmt(DdlStmt ddlStmt, String dbName, TableIf table, Lis
sb.append(olapTable.skipWriteIndexOnLoad()).append("\"");
}
+ // compaction policy
+ if (olapTable.getCompactionPolicy() != null && !olapTable.getCompactionPolicy().equals("")) {
+ sb.append(",\n\"").append(PropertyAnalyzer.PROPERTIES_COMPACTION_POLICY).append("\" = \"");
+ sb.append(olapTable.getCompactionPolicy()).append("\"");
+ }
+
+ // time series compaction goal size
+ if (olapTable.getTimeSeriesCompactionGoalSizeMbytes() != null
+ && olapTable.getTimeSeriesCompactionGoalSizeMbytes() != 512) {
+ sb.append(",\n\"").append(PropertyAnalyzer
+ .PROPERTIES_TIME_SERIES_COMPACTION_GOAL_SIZE_MBYTES).append("\" = \"");
+ sb.append(olapTable.getTimeSeriesCompactionGoalSizeMbytes()).append("\"");
+ }
+
+ // time series compaction file count threshold
+ if (olapTable.getTimeSeriesCompactionFileCountThreshold() != null
+ && olapTable.getTimeSeriesCompactionFileCountThreshold() != 2000) {
+ sb.append(",\n\"").append(PropertyAnalyzer
+ .PROPERTIES_TIME_SERIES_COMPACTION_FILE_COUNT_THRESHOLD).append("\" = \"");
+ sb.append(olapTable.getTimeSeriesCompactionFileCountThreshold()).append("\"");
+ }
+
+ // time series compaction time threshold
+ if (olapTable.getTimeSeriesCompactionTimeThresholdSeconds() != null
+ && olapTable.getTimeSeriesCompactionTimeThresholdSeconds() != 3600) {
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] csun5285 commented on a diff in pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
Posted by "csun5285 (via GitHub)" <gi...@apache.org>.
csun5285 commented on code in PR #22159:
URL: https://github.com/apache/doris/pull/22159#discussion_r1278307881
##########
be/src/olap/cumulative_compaction_policy.cpp:
##########
@@ -368,11 +368,13 @@ int64_t SizeBasedCumulativeCompactionPolicy::_level_size(const int64_t size) {
}
std::shared_ptr<CumulativeCompactionPolicy>
-CumulativeCompactionPolicyFactory::create_cumulative_compaction_policy() {
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] xiaokang commented on a diff in pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
Posted by "xiaokang (via GitHub)" <gi...@apache.org>.
xiaokang commented on code in PR #22159:
URL: https://github.com/apache/doris/pull/22159#discussion_r1278966598
##########
fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java:
##########
@@ -3120,6 +3120,40 @@ public static void getDdlStmt(DdlStmt ddlStmt, String dbName, TableIf table, Lis
sb.append(olapTable.skipWriteIndexOnLoad()).append("\"");
}
+ // compaction policy
+ if (olapTable.getCompactionPolicy() != null && !olapTable.getCompactionPolicy().equals("")
+ && !olapTable.getCompactionPolicy().equals(PropertyAnalyzer.SIZE_BASED_COMPACTION_POLICY)) {
+ sb.append(",\n\"").append(PropertyAnalyzer.PROPERTIES_COMPACTION_POLICY).append("\" = \"");
+ sb.append(olapTable.getCompactionPolicy()).append("\"");
+ }
+
+ // time series compaction goal size
+ if (olapTable.getTimeSeriesCompactionGoalSizeMbytes() != null
+ && olapTable.getTimeSeriesCompactionGoalSizeMbytes()
+ != PropertyAnalyzer.TIME_SERIES_COMPACTION_GOAL_SIZE_MBYTES_DEFAULT_VALUE) {
Review Comment:
show default value as normal
##########
gensrc/thrift/AgentService.thrift:
##########
@@ -142,6 +142,10 @@ struct TCreateTabletReq {
19: optional bool enable_unique_key_merge_on_write = false
20: optional i64 storage_policy_id
21: optional TBinlogConfig binlog_config
+ 22: optional string compaction_policy = "size_based"
+ 23: optional i64 time_series_compaction_goal_size_mbytes = 512
Review Comment:
default 1024
##########
fe/fe-core/src/main/java/org/apache/doris/analysis/ModifyTablePropertiesClause.java:
##########
@@ -142,6 +182,65 @@ public void analyze(Analyzer analyzer) throws AnalysisException {
|| properties.containsKey(PropertyAnalyzer.PROPERTIES_BINLOG_MAX_BYTES)
|| properties.containsKey(PropertyAnalyzer.PROPERTIES_BINLOG_MAX_HISTORY_NUMS)) {
// do nothing, will be alter in SchemaChangeHandler.updateBinlogConfig
+ } else if (properties.containsKey(PropertyAnalyzer.PROPERTIES_COMPACTION_POLICY)) {
+ String compactionPolicy = properties.getOrDefault(PropertyAnalyzer.PROPERTIES_COMPACTION_POLICY, "");
+ if (compactionPolicy != null
+ && !compactionPolicy.equals(PropertyAnalyzer.TIME_SERIES_COMPACTION_POLICY)
+ && !compactionPolicy.equals(PropertyAnalyzer.SIZE_BASED_COMPACTION_POLICY)) {
+ throw new AnalysisException(
+ "Table compaction policy only support for " + PropertyAnalyzer.TIME_SERIES_COMPACTION_POLICY
+ + " or " + PropertyAnalyzer.SIZE_BASED_COMPACTION_POLICY);
+ }
+ this.needTableStable = false;
+ setCompactionPolicy(compactionPolicy);
+ } else if (properties.containsKey(PropertyAnalyzer.PROPERTIES_TIME_SERIES_COMPACTION_GOAL_SIZE_MBYTES)) {
+ long goalSizeMbytes;
+ String goalSizeMbytesStr = properties
+ .get(PropertyAnalyzer.PROPERTIES_TIME_SERIES_COMPACTION_GOAL_SIZE_MBYTES);
+ try {
+ goalSizeMbytes = Long.parseLong(goalSizeMbytesStr);
+ if (goalSizeMbytes < 0) {
+ throw new AnalysisException("Invalid time_series_compaction_goal_size_mbytes format: "
+ + goalSizeMbytesStr);
+ }
+ } catch (NumberFormatException e) {
+ throw new AnalysisException("Invalid time_series_compaction_goal_size_mbytes format: "
+ + goalSizeMbytesStr);
+ }
+ this.needTableStable = false;
+ setTimeSeriesCompactionGoalSizeMbytes(goalSizeMbytes);
+ } else if (properties.containsKey(PropertyAnalyzer.PROPERTIES_TIME_SERIES_COMPACTION_FILE_COUNT_THRESHOLD)) {
+ long fileCountThreshold;
+ String fileCountThresholdStr = properties
+ .get(PropertyAnalyzer.PROPERTIES_TIME_SERIES_COMPACTION_FILE_COUNT_THRESHOLD);
+ try {
+ fileCountThreshold = Long.parseLong(fileCountThresholdStr);
+ if (fileCountThreshold < 0) {
Review Comment:
10
##########
be/src/olap/tablet_manager.cpp:
##########
@@ -729,7 +731,10 @@ TabletSharedPtr TabletManager::find_best_tablet_to_compaction(
continue;
}
}
-
+ auto cumulative_compaction_policy =
+ tablet_ptr->tablet_meta()->compaction_policy() == CUMULATIVE_TIME_SERIES_POLICY
Review Comment:
at(tablet_ptr->tablet_meta()->compaction_policy())
##########
fe/fe-core/src/main/java/org/apache/doris/analysis/ModifyTablePropertiesClause.java:
##########
@@ -142,6 +182,65 @@ public void analyze(Analyzer analyzer) throws AnalysisException {
|| properties.containsKey(PropertyAnalyzer.PROPERTIES_BINLOG_MAX_BYTES)
|| properties.containsKey(PropertyAnalyzer.PROPERTIES_BINLOG_MAX_HISTORY_NUMS)) {
// do nothing, will be alter in SchemaChangeHandler.updateBinlogConfig
+ } else if (properties.containsKey(PropertyAnalyzer.PROPERTIES_COMPACTION_POLICY)) {
+ String compactionPolicy = properties.getOrDefault(PropertyAnalyzer.PROPERTIES_COMPACTION_POLICY, "");
+ if (compactionPolicy != null
+ && !compactionPolicy.equals(PropertyAnalyzer.TIME_SERIES_COMPACTION_POLICY)
+ && !compactionPolicy.equals(PropertyAnalyzer.SIZE_BASED_COMPACTION_POLICY)) {
+ throw new AnalysisException(
+ "Table compaction policy only support for " + PropertyAnalyzer.TIME_SERIES_COMPACTION_POLICY
+ + " or " + PropertyAnalyzer.SIZE_BASED_COMPACTION_POLICY);
+ }
+ this.needTableStable = false;
+ setCompactionPolicy(compactionPolicy);
+ } else if (properties.containsKey(PropertyAnalyzer.PROPERTIES_TIME_SERIES_COMPACTION_GOAL_SIZE_MBYTES)) {
+ long goalSizeMbytes;
+ String goalSizeMbytesStr = properties
+ .get(PropertyAnalyzer.PROPERTIES_TIME_SERIES_COMPACTION_GOAL_SIZE_MBYTES);
+ try {
+ goalSizeMbytes = Long.parseLong(goalSizeMbytesStr);
+ if (goalSizeMbytes < 0) {
+ throw new AnalysisException("Invalid time_series_compaction_goal_size_mbytes format: "
+ + goalSizeMbytesStr);
+ }
+ } catch (NumberFormatException e) {
+ throw new AnalysisException("Invalid time_series_compaction_goal_size_mbytes format: "
+ + goalSizeMbytesStr);
+ }
+ this.needTableStable = false;
+ setTimeSeriesCompactionGoalSizeMbytes(goalSizeMbytes);
+ } else if (properties.containsKey(PropertyAnalyzer.PROPERTIES_TIME_SERIES_COMPACTION_FILE_COUNT_THRESHOLD)) {
+ long fileCountThreshold;
+ String fileCountThresholdStr = properties
+ .get(PropertyAnalyzer.PROPERTIES_TIME_SERIES_COMPACTION_FILE_COUNT_THRESHOLD);
+ try {
+ fileCountThreshold = Long.parseLong(fileCountThresholdStr);
+ if (fileCountThreshold < 0) {
+ throw new AnalysisException("Invalid time_series_compaction_file_count_threshold format: "
+ + fileCountThresholdStr);
+ }
+ } catch (NumberFormatException e) {
+ throw new AnalysisException("Invalid time_series_compaction_file_count_threshold format: "
+ + fileCountThresholdStr);
+ }
+ this.needTableStable = false;
+ setTimeSeriesCompactionFileCountThreshold(fileCountThreshold);
+ } else if (properties.containsKey(PropertyAnalyzer.PROPERTIES_TIME_SERIES_COMPACTION_TIME_THRESHOLD_SECONDS)) {
+ long timeThresholdSeconds;
+ String timeThresholdSecondsStr = properties
+ .get(PropertyAnalyzer.PROPERTIES_TIME_SERIES_COMPACTION_TIME_THRESHOLD_SECONDS);
+ try {
+ timeThresholdSeconds = Long.parseLong(timeThresholdSecondsStr);
+ if (timeThresholdSeconds < 0) {
Review Comment:
60s
##########
fe/fe-core/src/main/java/org/apache/doris/analysis/ModifyTablePropertiesClause.java:
##########
@@ -142,6 +182,65 @@ public void analyze(Analyzer analyzer) throws AnalysisException {
|| properties.containsKey(PropertyAnalyzer.PROPERTIES_BINLOG_MAX_BYTES)
|| properties.containsKey(PropertyAnalyzer.PROPERTIES_BINLOG_MAX_HISTORY_NUMS)) {
// do nothing, will be alter in SchemaChangeHandler.updateBinlogConfig
+ } else if (properties.containsKey(PropertyAnalyzer.PROPERTIES_COMPACTION_POLICY)) {
+ String compactionPolicy = properties.getOrDefault(PropertyAnalyzer.PROPERTIES_COMPACTION_POLICY, "");
+ if (compactionPolicy != null
+ && !compactionPolicy.equals(PropertyAnalyzer.TIME_SERIES_COMPACTION_POLICY)
+ && !compactionPolicy.equals(PropertyAnalyzer.SIZE_BASED_COMPACTION_POLICY)) {
+ throw new AnalysisException(
+ "Table compaction policy only support for " + PropertyAnalyzer.TIME_SERIES_COMPACTION_POLICY
+ + " or " + PropertyAnalyzer.SIZE_BASED_COMPACTION_POLICY);
+ }
+ this.needTableStable = false;
+ setCompactionPolicy(compactionPolicy);
+ } else if (properties.containsKey(PropertyAnalyzer.PROPERTIES_TIME_SERIES_COMPACTION_GOAL_SIZE_MBYTES)) {
+ long goalSizeMbytes;
+ String goalSizeMbytesStr = properties
+ .get(PropertyAnalyzer.PROPERTIES_TIME_SERIES_COMPACTION_GOAL_SIZE_MBYTES);
+ try {
+ goalSizeMbytes = Long.parseLong(goalSizeMbytesStr);
+ if (goalSizeMbytes < 0) {
Review Comment:
10MB
##########
be/src/olap/olap_server.cpp:
##########
@@ -976,8 +981,18 @@ Status StorageEngine::_submit_compaction_task(TabletSharedPtr tablet,
Status StorageEngine::submit_compaction_task(TabletSharedPtr tablet, CompactionType compaction_type,
bool force) {
_update_cumulative_compaction_policy();
- if (tablet->get_cumulative_compaction_policy() == nullptr) {
- tablet->set_cumulative_compaction_policy(_cumulative_compaction_policy);
+ // alter table tableName set ("compaction_policy"="time_series")
+ // if atler table's compaction policy, we need to modify tablet compaction policy shared ptr
+ if (tablet->get_cumulative_compaction_policy() == nullptr ||
+ tablet->get_cumulative_compaction_policy()->name() !=
+ tablet->tablet_meta()->compaction_policy()) {
+ if (tablet->tablet_meta()->compaction_policy() == CUMULATIVE_TIME_SERIES_POLICY) {
Review Comment:
at(tablet->tablet_meta()->compaction_policy())
##########
regression-test/suites/compaction/test_table_level_compaction_policy.groovy:
##########
@@ -0,0 +1,131 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+import org.codehaus.groovy.runtime.IOGroovyMethods
+
+suite("test_table_level_compaction_policy") {
+ def tableName = "test_table_level_compaction_policy"
+ sql """ DROP TABLE IF EXISTS ${tableName} """
+
+ sql """
+ CREATE TABLE ${tableName} (
+ `c_custkey` int(11) NOT NULL COMMENT "",
+ `c_name` varchar(26) NOT NULL COMMENT "",
+ `c_address` varchar(41) NOT NULL COMMENT "",
+ `c_city` varchar(11) NOT NULL COMMENT ""
+ )
+ DUPLICATE KEY (`c_custkey`)
+ DISTRIBUTED BY HASH(`c_custkey`) BUCKETS 1
+ PROPERTIES (
+ "replication_num" = "1",
+ "compaction_policy" = "time_series",
+ "time_series_compaction_goal_size_mbytes" = "1024",
+ "time_series_compaction_file_count_threshold" = "5000",
+ "time_series_compaction_time_threshold_seconds" = "86400"
+ );
+ """
+ result = sql """show create table ${tableName}"""
+ logger.info("${result}")
+ assertTrue(result.toString().containsIgnoreCase('"compaction_policy" = "time_series"'))
+ assertTrue(result.toString().containsIgnoreCase('"time_series_compaction_goal_size_mbytes" = "1024"'))
+ assertTrue(result.toString().containsIgnoreCase('"time_series_compaction_file_count_threshold" = "5000"'))
+ assertTrue(result.toString().containsIgnoreCase('"time_series_compaction_time_threshold_seconds" = "86400"'))
+
+ sql """
+ alter table ${tableName} set ("time_series_compaction_goal_size_mbytes" = "2048")
+ """
+
+ result = sql """show create table ${tableName}"""
+ logger.info("${result}")
+ assertTrue(result.toString().containsIgnoreCase('"time_series_compaction_goal_size_mbytes" = "2048"'))
+
+ sql """
+ alter table ${tableName} set ("time_series_compaction_file_count_threshold" = "6000")
+ """
+
+ result = sql """show create table ${tableName}"""
+ logger.info("${result}")
+ assertTrue(result.toString().containsIgnoreCase('"time_series_compaction_file_count_threshold" = "6000"'))
+
+ sql """
+ alter table ${tableName} set ("time_series_compaction_time_threshold_seconds" = "3000")
+ """
+
+ result = sql """show create table ${tableName}"""
+ logger.info("${result}")
+ assertTrue(result.toString().containsIgnoreCase('"time_series_compaction_time_threshold_seconds" = "3000"'))
+
+ sql """ DROP TABLE IF EXISTS ${tableName} """
+
+ test {
+ sql """
+ CREATE TABLE ${tableName} (
+ `c_custkey` int(11) NOT NULL COMMENT "",
+ `c_name` varchar(26) NOT NULL COMMENT "",
+ `c_address` varchar(41) NOT NULL COMMENT "",
+ `c_city` varchar(11) NOT NULL COMMENT ""
+ )
+ DUPLICATE KEY (`c_custkey`)
+ DISTRIBUTED BY HASH(`c_custkey`) BUCKETS 1
+ PROPERTIES (
+ "replication_num" = "1",
+ "compaction_policy" = "ok"
+ );
+ """
+ exception "compaction_policy must be time_series or size_based"
+ }
+
+ test {
+ sql """
+ CREATE TABLE ${tableName} (
+ `c_custkey` int(11) NOT NULL COMMENT "",
+ `c_name` varchar(26) NOT NULL COMMENT "",
+ `c_address` varchar(41) NOT NULL COMMENT "",
+ `c_city` varchar(11) NOT NULL COMMENT ""
+ )
+ DUPLICATE KEY (`c_custkey`)
+ DISTRIBUTED BY HASH(`c_custkey`) BUCKETS 1
+ PROPERTIES (
+ "replication_num" = "1",
+ "time_series_compaction_goal_size_mbytes" = "1024"
+ );
+ """
+ exception "only time series compaction policy support for time series config"
+ }
+
+ test {
+ sql """
+ CREATE TABLE ${tableName} (
+ `c_custkey` int(11) NOT NULL COMMENT "",
+ `c_name` varchar(26) NOT NULL COMMENT "",
+ `c_address` varchar(41) NOT NULL COMMENT "",
+ `c_city` varchar(11) NOT NULL COMMENT ""
+ )
+ DUPLICATE KEY (`c_custkey`)
+ DISTRIBUTED BY HASH(`c_custkey`) BUCKETS 1
+ PROPERTIES (
+ "replication_num" = "1"
+ );
+ """
+
+ sql """
+ alter table ${tableName} set ("compaction_policy" = "ok")
+ """
+ exception "Table compaction policy only support for time_series or size_based"
+ }
+ sql """ DROP TABLE IF EXISTS ${tableName} """
+}
Review Comment:
add a test case for no compaction_policy config and check its output
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] csun5285 commented on a diff in pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
Posted by "csun5285 (via GitHub)" <gi...@apache.org>.
csun5285 commented on code in PR #22159:
URL: https://github.com/apache/doris/pull/22159#discussion_r1279021089
##########
be/src/olap/olap_server.cpp:
##########
@@ -976,8 +981,18 @@ Status StorageEngine::_submit_compaction_task(TabletSharedPtr tablet,
Status StorageEngine::submit_compaction_task(TabletSharedPtr tablet, CompactionType compaction_type,
bool force) {
_update_cumulative_compaction_policy();
- if (tablet->get_cumulative_compaction_policy() == nullptr) {
- tablet->set_cumulative_compaction_policy(_cumulative_compaction_policy);
+ // alter table tableName set ("compaction_policy"="time_series")
+ // if atler table's compaction policy, we need to modify tablet compaction policy shared ptr
+ if (tablet->get_cumulative_compaction_policy() == nullptr ||
+ tablet->get_cumulative_compaction_policy()->name() !=
+ tablet->tablet_meta()->compaction_policy()) {
+ if (tablet->tablet_meta()->compaction_policy() == CUMULATIVE_TIME_SERIES_POLICY) {
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] csun5285 commented on pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
Posted by "csun5285 (via GitHub)" <gi...@apache.org>.
csun5285 commented on PR #22159:
URL: https://github.com/apache/doris/pull/22159#issuecomment-1658240092
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] csun5285 commented on a diff in pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
Posted by "csun5285 (via GitHub)" <gi...@apache.org>.
csun5285 commented on code in PR #22159:
URL: https://github.com/apache/doris/pull/22159#discussion_r1278308021
##########
be/src/olap/olap_server.cpp:
##########
@@ -876,9 +877,12 @@ std::vector<TabletSharedPtr> StorageEngine::_generate_compaction_tasks(
}
void StorageEngine::_update_cumulative_compaction_policy() {
- if (_cumulative_compaction_policy == nullptr) {
- _cumulative_compaction_policy =
- CumulativeCompactionPolicyFactory::create_cumulative_compaction_policy();
+ if (_all_cumulative_compaction_policy.size() < 2) {
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] csun5285 commented on a diff in pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
Posted by "csun5285 (via GitHub)" <gi...@apache.org>.
csun5285 commented on code in PR #22159:
URL: https://github.com/apache/doris/pull/22159#discussion_r1279015472
##########
fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java:
##########
@@ -3120,6 +3120,40 @@ public static void getDdlStmt(DdlStmt ddlStmt, String dbName, TableIf table, Lis
sb.append(olapTable.skipWriteIndexOnLoad()).append("\"");
}
+ // compaction policy
+ if (olapTable.getCompactionPolicy() != null && !olapTable.getCompactionPolicy().equals("")
+ && !olapTable.getCompactionPolicy().equals(PropertyAnalyzer.SIZE_BASED_COMPACTION_POLICY)) {
+ sb.append(",\n\"").append(PropertyAnalyzer.PROPERTIES_COMPACTION_POLICY).append("\" = \"");
+ sb.append(olapTable.getCompactionPolicy()).append("\"");
+ }
+
+ // time series compaction goal size
+ if (olapTable.getTimeSeriesCompactionGoalSizeMbytes() != null
+ && olapTable.getTimeSeriesCompactionGoalSizeMbytes()
+ != PropertyAnalyzer.TIME_SERIES_COMPACTION_GOAL_SIZE_MBYTES_DEFAULT_VALUE) {
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #22159:
URL: https://github.com/apache/doris/pull/22159#issuecomment-1658203244
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 48.2 seconds
stream load tsv: 510 seconds loaded 74807831229 Bytes, about 139 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.6 seconds inserted 10000000 Rows, about 337K ops/s
storage size: 17162160055 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #22159:
URL: https://github.com/apache/doris/pull/22159#issuecomment-1657047480
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] csun5285 commented on a diff in pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
Posted by "csun5285 (via GitHub)" <gi...@apache.org>.
csun5285 commented on code in PR #22159:
URL: https://github.com/apache/doris/pull/22159#discussion_r1278509831
##########
be/src/agent/task_worker_pool.cpp:
##########
@@ -433,6 +433,63 @@ void TaskWorkerPool::_update_tablet_meta_worker_thread_callback() {
tablet->tablet_schema_unlocked()->set_is_in_memory(tablet_meta_info.is_in_memory);
need_to_save = true;
}
+ if (tablet_meta_info.__isset.compaction_policy) {
+ tablet->tablet_meta()->mutable_tablet_schema()->set_compaction_policy(
+ tablet_meta_info.compaction_policy);
+ std::shared_lock rlock(tablet->get_header_lock());
+ for (auto& rowset_meta : tablet->tablet_meta()->all_mutable_rs_metas()) {
+ rowset_meta->tablet_schema()->set_compaction_policy(
+ tablet_meta_info.compaction_policy);
+ }
+ tablet->tablet_schema_unlocked()->set_compaction_policy(
Review Comment:
now I've moved them to the tablet meta
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] csun5285 commented on pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
Posted by "csun5285 (via GitHub)" <gi...@apache.org>.
csun5285 commented on PR #22159:
URL: https://github.com/apache/doris/pull/22159#issuecomment-1657046034
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] csun5285 commented on pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
Posted by "csun5285 (via GitHub)" <gi...@apache.org>.
csun5285 commented on PR #22159:
URL: https://github.com/apache/doris/pull/22159#issuecomment-1659578095
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #22159:
URL: https://github.com/apache/doris/pull/22159#issuecomment-1651754508
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.71 seconds
stream load tsv: 508 seconds loaded 74807831229 Bytes, about 140 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.3 seconds inserted 10000000 Rows, about 341K ops/s
storage size: 17162378898 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] csun5285 commented on pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
Posted by "csun5285 (via GitHub)" <gi...@apache.org>.
csun5285 commented on PR #22159:
URL: https://github.com/apache/doris/pull/22159#issuecomment-1652840498
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #22159:
URL: https://github.com/apache/doris/pull/22159#issuecomment-1647493243
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] csun5285 commented on a diff in pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
Posted by "csun5285 (via GitHub)" <gi...@apache.org>.
csun5285 commented on code in PR #22159:
URL: https://github.com/apache/doris/pull/22159#discussion_r1279003905
##########
fe/fe-core/src/main/java/org/apache/doris/analysis/ModifyTablePropertiesClause.java:
##########
@@ -142,6 +182,65 @@ public void analyze(Analyzer analyzer) throws AnalysisException {
|| properties.containsKey(PropertyAnalyzer.PROPERTIES_BINLOG_MAX_BYTES)
|| properties.containsKey(PropertyAnalyzer.PROPERTIES_BINLOG_MAX_HISTORY_NUMS)) {
// do nothing, will be alter in SchemaChangeHandler.updateBinlogConfig
+ } else if (properties.containsKey(PropertyAnalyzer.PROPERTIES_COMPACTION_POLICY)) {
+ String compactionPolicy = properties.getOrDefault(PropertyAnalyzer.PROPERTIES_COMPACTION_POLICY, "");
+ if (compactionPolicy != null
+ && !compactionPolicy.equals(PropertyAnalyzer.TIME_SERIES_COMPACTION_POLICY)
+ && !compactionPolicy.equals(PropertyAnalyzer.SIZE_BASED_COMPACTION_POLICY)) {
+ throw new AnalysisException(
+ "Table compaction policy only support for " + PropertyAnalyzer.TIME_SERIES_COMPACTION_POLICY
+ + " or " + PropertyAnalyzer.SIZE_BASED_COMPACTION_POLICY);
+ }
+ this.needTableStable = false;
+ setCompactionPolicy(compactionPolicy);
+ } else if (properties.containsKey(PropertyAnalyzer.PROPERTIES_TIME_SERIES_COMPACTION_GOAL_SIZE_MBYTES)) {
+ long goalSizeMbytes;
+ String goalSizeMbytesStr = properties
+ .get(PropertyAnalyzer.PROPERTIES_TIME_SERIES_COMPACTION_GOAL_SIZE_MBYTES);
+ try {
+ goalSizeMbytes = Long.parseLong(goalSizeMbytesStr);
+ if (goalSizeMbytes < 0) {
Review Comment:
done
##########
fe/fe-core/src/main/java/org/apache/doris/analysis/ModifyTablePropertiesClause.java:
##########
@@ -142,6 +182,65 @@ public void analyze(Analyzer analyzer) throws AnalysisException {
|| properties.containsKey(PropertyAnalyzer.PROPERTIES_BINLOG_MAX_BYTES)
|| properties.containsKey(PropertyAnalyzer.PROPERTIES_BINLOG_MAX_HISTORY_NUMS)) {
// do nothing, will be alter in SchemaChangeHandler.updateBinlogConfig
+ } else if (properties.containsKey(PropertyAnalyzer.PROPERTIES_COMPACTION_POLICY)) {
+ String compactionPolicy = properties.getOrDefault(PropertyAnalyzer.PROPERTIES_COMPACTION_POLICY, "");
+ if (compactionPolicy != null
+ && !compactionPolicy.equals(PropertyAnalyzer.TIME_SERIES_COMPACTION_POLICY)
+ && !compactionPolicy.equals(PropertyAnalyzer.SIZE_BASED_COMPACTION_POLICY)) {
+ throw new AnalysisException(
+ "Table compaction policy only support for " + PropertyAnalyzer.TIME_SERIES_COMPACTION_POLICY
+ + " or " + PropertyAnalyzer.SIZE_BASED_COMPACTION_POLICY);
+ }
+ this.needTableStable = false;
+ setCompactionPolicy(compactionPolicy);
+ } else if (properties.containsKey(PropertyAnalyzer.PROPERTIES_TIME_SERIES_COMPACTION_GOAL_SIZE_MBYTES)) {
+ long goalSizeMbytes;
+ String goalSizeMbytesStr = properties
+ .get(PropertyAnalyzer.PROPERTIES_TIME_SERIES_COMPACTION_GOAL_SIZE_MBYTES);
+ try {
+ goalSizeMbytes = Long.parseLong(goalSizeMbytesStr);
+ if (goalSizeMbytes < 0) {
+ throw new AnalysisException("Invalid time_series_compaction_goal_size_mbytes format: "
+ + goalSizeMbytesStr);
+ }
+ } catch (NumberFormatException e) {
+ throw new AnalysisException("Invalid time_series_compaction_goal_size_mbytes format: "
+ + goalSizeMbytesStr);
+ }
+ this.needTableStable = false;
+ setTimeSeriesCompactionGoalSizeMbytes(goalSizeMbytes);
+ } else if (properties.containsKey(PropertyAnalyzer.PROPERTIES_TIME_SERIES_COMPACTION_FILE_COUNT_THRESHOLD)) {
+ long fileCountThreshold;
+ String fileCountThresholdStr = properties
+ .get(PropertyAnalyzer.PROPERTIES_TIME_SERIES_COMPACTION_FILE_COUNT_THRESHOLD);
+ try {
+ fileCountThreshold = Long.parseLong(fileCountThresholdStr);
+ if (fileCountThreshold < 0) {
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #22159:
URL: https://github.com/apache/doris/pull/22159#issuecomment-1658300471
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 48.6 seconds
stream load tsv: 515 seconds loaded 74807831229 Bytes, about 138 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 30 seconds loaded 861443392 Bytes, about 27 MB/s
insert into select: 29.4 seconds inserted 10000000 Rows, about 340K ops/s
storage size: 17162134404 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #22159:
URL: https://github.com/apache/doris/pull/22159#issuecomment-1657360266
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.94 seconds
stream load tsv: 504 seconds loaded 74807831229 Bytes, about 141 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 30 seconds loaded 861443392 Bytes, about 27 MB/s
insert into select: 30.0 seconds inserted 10000000 Rows, about 333K ops/s
storage size: 17170413756 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #22159:
URL: https://github.com/apache/doris/pull/22159#issuecomment-1659582357
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #22159:
URL: https://github.com/apache/doris/pull/22159#issuecomment-1658071288
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] csun5285 commented on a diff in pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
Posted by "csun5285 (via GitHub)" <gi...@apache.org>.
csun5285 commented on code in PR #22159:
URL: https://github.com/apache/doris/pull/22159#discussion_r1279023738
##########
regression-test/suites/compaction/test_table_level_compaction_policy.groovy:
##########
@@ -0,0 +1,131 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+import org.codehaus.groovy.runtime.IOGroovyMethods
+
+suite("test_table_level_compaction_policy") {
+ def tableName = "test_table_level_compaction_policy"
+ sql """ DROP TABLE IF EXISTS ${tableName} """
+
+ sql """
+ CREATE TABLE ${tableName} (
+ `c_custkey` int(11) NOT NULL COMMENT "",
+ `c_name` varchar(26) NOT NULL COMMENT "",
+ `c_address` varchar(41) NOT NULL COMMENT "",
+ `c_city` varchar(11) NOT NULL COMMENT ""
+ )
+ DUPLICATE KEY (`c_custkey`)
+ DISTRIBUTED BY HASH(`c_custkey`) BUCKETS 1
+ PROPERTIES (
+ "replication_num" = "1",
+ "compaction_policy" = "time_series",
+ "time_series_compaction_goal_size_mbytes" = "1024",
+ "time_series_compaction_file_count_threshold" = "5000",
+ "time_series_compaction_time_threshold_seconds" = "86400"
+ );
+ """
+ result = sql """show create table ${tableName}"""
+ logger.info("${result}")
+ assertTrue(result.toString().containsIgnoreCase('"compaction_policy" = "time_series"'))
+ assertTrue(result.toString().containsIgnoreCase('"time_series_compaction_goal_size_mbytes" = "1024"'))
+ assertTrue(result.toString().containsIgnoreCase('"time_series_compaction_file_count_threshold" = "5000"'))
+ assertTrue(result.toString().containsIgnoreCase('"time_series_compaction_time_threshold_seconds" = "86400"'))
+
+ sql """
+ alter table ${tableName} set ("time_series_compaction_goal_size_mbytes" = "2048")
+ """
+
+ result = sql """show create table ${tableName}"""
+ logger.info("${result}")
+ assertTrue(result.toString().containsIgnoreCase('"time_series_compaction_goal_size_mbytes" = "2048"'))
+
+ sql """
+ alter table ${tableName} set ("time_series_compaction_file_count_threshold" = "6000")
+ """
+
+ result = sql """show create table ${tableName}"""
+ logger.info("${result}")
+ assertTrue(result.toString().containsIgnoreCase('"time_series_compaction_file_count_threshold" = "6000"'))
+
+ sql """
+ alter table ${tableName} set ("time_series_compaction_time_threshold_seconds" = "3000")
+ """
+
+ result = sql """show create table ${tableName}"""
+ logger.info("${result}")
+ assertTrue(result.toString().containsIgnoreCase('"time_series_compaction_time_threshold_seconds" = "3000"'))
+
+ sql """ DROP TABLE IF EXISTS ${tableName} """
+
+ test {
+ sql """
+ CREATE TABLE ${tableName} (
+ `c_custkey` int(11) NOT NULL COMMENT "",
+ `c_name` varchar(26) NOT NULL COMMENT "",
+ `c_address` varchar(41) NOT NULL COMMENT "",
+ `c_city` varchar(11) NOT NULL COMMENT ""
+ )
+ DUPLICATE KEY (`c_custkey`)
+ DISTRIBUTED BY HASH(`c_custkey`) BUCKETS 1
+ PROPERTIES (
+ "replication_num" = "1",
+ "compaction_policy" = "ok"
+ );
+ """
+ exception "compaction_policy must be time_series or size_based"
+ }
+
+ test {
+ sql """
+ CREATE TABLE ${tableName} (
+ `c_custkey` int(11) NOT NULL COMMENT "",
+ `c_name` varchar(26) NOT NULL COMMENT "",
+ `c_address` varchar(41) NOT NULL COMMENT "",
+ `c_city` varchar(11) NOT NULL COMMENT ""
+ )
+ DUPLICATE KEY (`c_custkey`)
+ DISTRIBUTED BY HASH(`c_custkey`) BUCKETS 1
+ PROPERTIES (
+ "replication_num" = "1",
+ "time_series_compaction_goal_size_mbytes" = "1024"
+ );
+ """
+ exception "only time series compaction policy support for time series config"
+ }
+
+ test {
+ sql """
+ CREATE TABLE ${tableName} (
+ `c_custkey` int(11) NOT NULL COMMENT "",
+ `c_name` varchar(26) NOT NULL COMMENT "",
+ `c_address` varchar(41) NOT NULL COMMENT "",
+ `c_city` varchar(11) NOT NULL COMMENT ""
+ )
+ DUPLICATE KEY (`c_custkey`)
+ DISTRIBUTED BY HASH(`c_custkey`) BUCKETS 1
+ PROPERTIES (
+ "replication_num" = "1"
+ );
+ """
+
+ sql """
+ alter table ${tableName} set ("compaction_policy" = "ok")
+ """
+ exception "Table compaction policy only support for time_series or size_based"
+ }
+ sql """ DROP TABLE IF EXISTS ${tableName} """
+}
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #22159:
URL: https://github.com/apache/doris/pull/22159#issuecomment-1651577699
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] csun5285 commented on a diff in pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
Posted by "csun5285 (via GitHub)" <gi...@apache.org>.
csun5285 commented on code in PR #22159:
URL: https://github.com/apache/doris/pull/22159#discussion_r1278309282
##########
fe/fe-core/src/main/java/org/apache/doris/alter/SchemaChangeHandler.java:
##########
@@ -2137,13 +2137,40 @@ public void updateTableProperties(Database db, String tableName, Map<String, Str
}
long storagePolicyId = storagePolicyNameToId(storagePolicy);
- if (isInMemory < 0 && storagePolicyId < 0) {
- LOG.info("Properties already up-to-date");
- return;
+ // if (isInMemory < 0 && storagePolicyId < 0) {
Review Comment:
I have now added four properties, including code to check if any of these properties have changed before executing a schema change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] csun5285 commented on a diff in pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
Posted by "csun5285 (via GitHub)" <gi...@apache.org>.
csun5285 commented on code in PR #22159:
URL: https://github.com/apache/doris/pull/22159#discussion_r1278509717
##########
be/src/agent/task_worker_pool.cpp:
##########
@@ -433,6 +433,63 @@ void TaskWorkerPool::_update_tablet_meta_worker_thread_callback() {
tablet->tablet_schema_unlocked()->set_is_in_memory(tablet_meta_info.is_in_memory);
need_to_save = true;
}
+ if (tablet_meta_info.__isset.compaction_policy) {
+ tablet->tablet_meta()->mutable_tablet_schema()->set_compaction_policy(
+ tablet_meta_info.compaction_policy);
+ std::shared_lock rlock(tablet->get_header_lock());
Review Comment:
the compaction-related properties were previously stored in the tablet schema, and now I've moved them to the tablet meta
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #22159:
URL: https://github.com/apache/doris/pull/22159#issuecomment-1657044981
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] csun5285 commented on pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
Posted by "csun5285 (via GitHub)" <gi...@apache.org>.
csun5285 commented on PR #22159:
URL: https://github.com/apache/doris/pull/22159#issuecomment-1657044837
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] csun5285 commented on pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
Posted by "csun5285 (via GitHub)" <gi...@apache.org>.
csun5285 commented on PR #22159:
URL: https://github.com/apache/doris/pull/22159#issuecomment-1657097417
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org