You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "csun5285 (via GitHub)" <gi...@apache.org> on 2026/05/25 11:16:09 UTC
[PR] [refactor](BE) split EncodingInfo defaults into 4 explicit maps [doris]
csun5285 opened a new pull request, #63622:
URL: https://github.com/apache/doris/pull/63622
Replace the EncodingPreference + runtime hook machinery in EncodingInfoResolver with four explicit maps and four matching get methods:
- _legacy_default_map -> get_legacy_default_encoding(type)
- _v3_default_map -> get_v3_default_encoding(type)
- _value_seek_default_map -> get_value_seek_encoding(type)
- _encoding_map -> get(type, encoding, out)
No on-disk format change; the resolved encodings written into ColumnMetaPB match the pre-refactor outputs for both legacy and V3 tablets.
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](BE) split EncodingInfo defaults into 4 explicit maps [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #63622:
URL: https://github.com/apache/doris/pull/63622#discussion_r3301217630
##########
be/src/storage/segment/vertical_segment_writer.cpp:
##########
@@ -160,11 +161,14 @@ VerticalSegmentWriter::~VerticalSegmentWriter() {
}
void VerticalSegmentWriter::_init_column_meta(ColumnMetaPB* meta, uint32_t column_id,
Review Comment:
这个vertical 里保存了，tablet schema，为啥还要传递这种use v3 encoding 的变量
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](BE) split EncodingInfo defaults into 4 explicit maps [doris]
Posted by "csun5285 (via GitHub)" <gi...@apache.org>.
csun5285 commented on PR #63622:
URL: https://github.com/apache/doris/pull/63622#issuecomment-4535113008
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](BE) split EncodingInfo defaults into 4 explicit maps [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63622:
URL: https://github.com/apache/doris/pull/63622#issuecomment-4545663614
<details>
<summary>TPC-H: <b>Total hot run time: 31135 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7e8f8df1fe781d0834691c41471270666298b9cb, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17718 3985 4012 3985
q2 q3 10774 1394 815 815
q4 4720 475 351 351
q5 8335 2287 2083 2083
q6 393 180 141 141
q7 954 763 652 652
q8 9384 1705 1595 1595
q9 7093 4999 4926 4926
q10 6488 2264 1884 1884
q11 450 275 242 242
q12 693 442 306 306
q13 18162 3465 2839 2839
q14 269 256 236 236
q15 q16 827 796 714 714
q17 968 906 988 906
q18 7024 5721 5493 5493
q19 1195 1263 989 989
q20 503 414 266 266
q21 5717 2558 2411 2411
q22 437 361 301 301
Total cold run time: 102104 ms
Total hot run time: 31135 ms
----- Round 2, with runtime_filter_mode=off -----
workload_group 0
============================================
show table status;
Name Engine Version Row_format Rows Avg_row_length Data_length Max_data_length Index_length Data_free Auto_increment Create_time Update_time Check_time Collation Checksum Create_options Comment
revenue0 View NULL NULL -1 0 0 NULL 0 NULL NULL 2023-12-26 18:27:24 NULL NULL utf-8 NULL NULL
partsupp Doris NULL NULL 80000000 56 4534120086 NULL 44625495 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:44:20 NULL utf-8 NULL NULL
part Doris NULL NULL 20000000 37 748811035 NULL 1935627 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:27:57 NULL utf-8 NULL NULL
nation Doris NULL NULL 25 138 3473 NULL 366 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:27:24 NULL utf-8 NULL NULL
customer Doris NULL NULL 15000000 92 1381653732 NULL 4374759 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:27:43 NULL utf-8 NULL NULL
lineitem Doris NULL NULL 600037902 33 19843441616 NULL 61784740 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:38:59 NULL utf-8 NULL NULL
supplier Doris NULL NULL 1000000 87 87519212 NULL 194931 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:27:25 NULL utf-8 NULL NULL
region Doris NULL NULL 5 240 1201 NULL 147 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:27:24 NULL utf-8 NULL NULL
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4337 4260 4352 4260
q2 q3 4525 4883 4368 4368
q4 2114 2238 1399 1399
q5 4434 4325 4915 4325
q6 257 203 145 145
q7 1978 1859 1616 1616
q8 2782 2201 2239 2201
q9 8127 8040 8028 8028
q10 4837 4647 q11 q12 q13 q14 q15 q16 q17 q18 q19 q20 q21 q22 Total cold run time: 28554 ms
Total hot run time: 26342 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](BE) split EncodingInfo defaults into 4 explicit maps [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63622:
URL: https://github.com/apache/doris/pull/63622#issuecomment-4560895279
# FE UT Coverage Report
Increment line coverage ` 50.00% (1/2)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63622_ed0bea85afb9255428029d24f501c0b2c9b97585/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63622_ed0bea85afb9255428029d24f501c0b2c9b97585/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](BE) split EncodingInfo defaults into 4 explicit maps [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #63622:
URL: https://github.com/apache/doris/pull/63622#discussion_r3308234107
##########
be/src/storage/segment/encoding_info.cpp:
##########
@@ -214,124 +216,181 @@ struct TypeEncodingTraits<type, PREFIX_ENCODING, Slice> {
};
EncodingInfoResolver::EncodingInfoResolver() {
- _add_map<FieldType::OLAP_FIELD_TYPE_TINYINT, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_TINYINT, FOR_ENCODING, true>();
- _add_map<FieldType::OLAP_FIELD_TYPE_TINYINT, PLAIN_ENCODING>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_SMALLINT, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_SMALLINT, FOR_ENCODING, true>();
- _add_map<FieldType::OLAP_FIELD_TYPE_SMALLINT, PLAIN_ENCODING>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_INT, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_INT, FOR_ENCODING, true>();
- _add_map<FieldType::OLAP_FIELD_TYPE_INT, PLAIN_ENCODING>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_BIGINT, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_BIGINT, FOR_ENCODING, true>();
- _add_map<FieldType::OLAP_FIELD_TYPE_BIGINT, PLAIN_ENCODING>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_UNSIGNED_BIGINT, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_UNSIGNED_INT, BIT_SHUFFLE>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_LARGEINT, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_LARGEINT, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_LARGEINT, FOR_ENCODING, true>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_FLOAT, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_FLOAT, PLAIN_ENCODING>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_DOUBLE, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_DOUBLE, PLAIN_ENCODING>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_CHAR, DICT_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_CHAR, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_CHAR, PLAIN_ENCODING_V2>();
- _add_map<FieldType::OLAP_FIELD_TYPE_CHAR, PREFIX_ENCODING, true>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_VARCHAR, DICT_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_VARCHAR, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_VARCHAR, PLAIN_ENCODING_V2>();
- _add_map<FieldType::OLAP_FIELD_TYPE_VARCHAR, PREFIX_ENCODING, true>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_STRING, DICT_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_STRING, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_STRING, PLAIN_ENCODING_V2>();
- _add_map<FieldType::OLAP_FIELD_TYPE_STRING, PREFIX_ENCODING, true>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_JSONB, DICT_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_JSONB, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_JSONB, PLAIN_ENCODING_V2>();
- _add_map<FieldType::OLAP_FIELD_TYPE_JSONB, PREFIX_ENCODING, true>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_VARIANT, DICT_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_VARIANT, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_VARIANT, PLAIN_ENCODING_V2>();
- _add_map<FieldType::OLAP_FIELD_TYPE_VARIANT, PREFIX_ENCODING, true>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_BOOL, RLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_BOOL, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_BOOL, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_BOOL, PLAIN_ENCODING, true>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_DATE, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_DATE, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_DATE, FOR_ENCODING, true>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_DATEV2, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_DATEV2, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_DATEV2, FOR_ENCODING, true>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_DATETIMEV2, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_DATETIMEV2, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_DATETIMEV2, FOR_ENCODING, true>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_DATETIME, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_DATETIME, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_DATETIME, FOR_ENCODING, true>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_TIMESTAMPTZ, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_TIMESTAMPTZ, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_TIMESTAMPTZ, FOR_ENCODING, true>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_DECIMAL, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_DECIMAL, PLAIN_ENCODING>();
Review Comment:
所有的decimal 类型，或者底层是float 或者 int 类型表示的，是否都应该换成plan encoding？ 另外，为什么是plain encoding，而不是RLE？ 可以看看parquet的
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](BE) split EncodingInfo defaults into 4 explicit maps [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #63622:
URL: https://github.com/apache/doris/pull/63622#discussion_r3303380059
##########
be/src/storage/segment/encoding_info.h:
##########
@@ -107,21 +111,40 @@ class EncodingInfoResolver {
EncodingInfoResolver();
~EncodingInfoResolver();
- EncodingTypePB get_default_encoding(FieldType type, EncodingPreference encoding_preference,
- bool optimize_value_seek) const;
+ EncodingTypePB get_legacy_default_encoding(FieldType type) const;
+ EncodingTypePB get_v3_default_encoding(FieldType type) const;
+ EncodingTypePB get_value_seek_encoding(FieldType type) const;
- Status get(FieldType data_type, EncodingTypePB encoding_type,
- EncodingPreference encoding_preference, const EncodingInfo** out);
+ Status get(FieldType data_type, EncodingTypePB encoding_type, const EncodingInfo** out);
private:
- // Not thread-safe
- template <FieldType type, EncodingTypePB encoding_type, bool optimize_value_seek = false>
- void _add_map();
-
- std::unordered_map<FieldType, EncodingTypePB, EncodingMapHash> _default_encoding_type_map;
+ // Registration helpers used by the constructor. Not thread-safe.
+ //
+ // _add: this (type, encoding) is a supported combination.
+ // _set_legacy_default: ... and is the default for the legacy (V1/V2) write path.
+ // _set_v3_default: ... and is the default for the V3 write path.
+ // _set_value_seek_default: ... and is the default for the value-seek path (PK / zone map).
+ //
+ // Each "_set_*_default" inserts into _encoding_map (idempotently) and then writes the
+ // corresponding default map exactly once (enforced by DCHECK).
+ template <FieldType type, EncodingTypePB encoding>
+ void _add();
+ template <FieldType type, EncodingTypePB encoding>
+ void _set_legacy_default();
+ template <FieldType type, EncodingTypePB encoding>
+ void _set_v3_default();
+ template <FieldType type, EncodingTypePB encoding>
+ void _set_value_seek_default();
+
+ static EncodingTypePB _lookup(
+ const std::unordered_map<FieldType, EncodingTypePB, EncodingMapHash>& m, FieldType t) {
+ auto it = m.find(t);
+ return it != m.end() ? it->second : UNKNOWN_ENCODING;
+ }
- // default encoding for each type which optimizes value seek
- std::unordered_map<FieldType, EncodingTypePB, EncodingMapHash> _value_seek_encoding_map;
+ std::unordered_map<FieldType, EncodingTypePB, EncodingMapHash> _legacy_default_map;
+ std::unordered_map<FieldType, EncodingTypePB, EncodingMapHash> _v3_default_map;
+ std::unordered_map<FieldType, EncodingTypePB, EncodingMapHash> _value_seek_default_map;
Review Comment:
这个value seek 想想换个名字啊
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](BE) split EncodingInfo defaults into 4 explicit maps [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63622:
URL: https://github.com/apache/doris/pull/63622#issuecomment-4545731004
<details>
<summary>TPC-DS: <b>Total hot run time: 76522 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7e8f8df1fe781d0834691c41471270666298b9cb, data reload: false
query5 4313 672 512 512
query6 347 227 201 201
query7 4280 584 305 305
query8 328 237 221 221
query9 8849 4205 4178 4178
query10 452 336 297 297
query11 5815 2582 2265 2265
query12 172 129 126 126
query13 1272 633 457 457
query14 6179 5470 5175 5175
query14_1 4583 4514 4499 4499
query15 212 206 193 193
query16 1045 457 420 420
query17 1037 737 604 604
query18 2478 490 365 365
query19 238 195 165 165
query20 139 131 129 129
query21 211 137 115 115
query22 13737 13765 13394 13394
query23 17329 16793 16483 16483
query23_1 16564 16472 16610 16472
query24 7561 1833 1404 1404
query24_1 1435 1412 1379 1379
query25 595 517 434 434
query26 976 326 181 181
query27 3212
query28
query29
query30
query31
query32
query33
query34
query35
query36
query37
query38
query39
query39_1
query40
query41
query42
query43
query44
query45
query46
query47
query48
query49
query50
query51
query52
query53
query54
query55
query56
query57
query58
query59
query60
query61
query62
query63
query64
query65
query66
query67
query68
query69
query70
query71
query72
query73
query74
query75
query76
query77
query78
query79
query80
query81
query82
query83
query84
query85
query86
query87
query88
query89
query90
query91
query92
query93
query94
query95
query96
query97
query98
query99
Total cold run time: 133039 ms
Total hot run time: 76522 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](BE) split EncodingInfo defaults into 4 explicit maps [doris]
Posted by "csun5285 (via GitHub)" <gi...@apache.org>.
csun5285 commented on PR #63622:
URL: https://github.com/apache/doris/pull/63622#issuecomment-4544749264
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](BE) split EncodingInfo defaults into 4 explicit maps [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #63622:
URL: https://github.com/apache/doris/pull/63622#discussion_r3308850854
##########
be/src/storage/tablet/tablet_meta.cpp:
##########
@@ -264,21 +264,11 @@ TabletMeta::TabletMeta(int64_t table_id, int64_t partition_id, int64_t tablet_id
case TStorageFormat::V1:
break;
case TStorageFormat::V3:
- schema_pb_for_data->set_is_external_segment_column_meta_used(true);
- _schema->set_external_segment_meta_used_default(true);
-
- schema_pb_for_data->set_integer_type_default_use_plain_encoding(true);
- _schema->set_integer_type_default_use_plain_encoding(true);
- schema_pb_for_data->set_binary_plain_encoding_default_impl(
- BinaryPlainEncodingTypePB::BINARY_PLAIN_ENCODING_V2);
- _schema->set_binary_plain_encoding_default_impl(
- BinaryPlainEncodingTypePB::BINARY_PLAIN_ENCODING_V2);
-
+ schema_pb_for_data->set_storage_format(TabletStorageFormatPB::TABLET_STORAGE_FORMAT_V3);
Review Comment:
This creates a V3 tablet schema by writing only the new `storage_format` field. That is not compatible with older BEs that do not know field 36: they ignore it and fall back to the legacy flags (`is_external_segment_column_meta_used`, `integer_type_default_use_plain_encoding`, `binary_plain_encoding_default_impl`), which this path now leaves unset. In a rolling upgrade/rollback or any mixed-version consumption of persisted `TabletSchemaPB`, such a tablet is interpreted as V2 and new segments can be written/read with the wrong footer/encoding policy. The cloud FE path still writes both representations for backward compatibility, so the non-cloud BE create path should also keep setting the three legacy V3 flags when `storage_format == V3` while adding the new field.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](BE) split EncodingInfo defaults into 4 explicit maps [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63622:
URL: https://github.com/apache/doris/pull/63622#issuecomment-4562678234
# FE Regression Coverage Report
Increment line coverage ` 3.17% (2/63)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63622_8eff36afaba543f4d0dca6550f163b964dd15be9_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63622_8eff36afaba543f4d0dca6550f163b964dd15be9_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](BE) split EncodingInfo defaults into 4 explicit maps [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #63622:
URL: https://github.com/apache/doris/pull/63622#discussion_r3302051025
##########
be/src/storage/segment/segment_writer.cpp:
##########
@@ -146,11 +147,13 @@ SegmentWriter::~SegmentWriter() {
}
void SegmentWriter::init_column_meta(ColumnMetaPB* meta, uint32_t column_id,
- const TabletColumn& column, TabletSchemaSPtr tablet_schema) {
+ const TabletColumn& column, bool use_v3_default_encoding) {
Review Comment:
我觉得还是直接传递tablet schema 更好一些，未来我们如果再给schema 加开关或者属性，不能一直扩参数列表
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](BE) split EncodingInfo defaults into 4 explicit maps [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63622:
URL: https://github.com/apache/doris/pull/63622#issuecomment-4561534004
<details>
<summary>TPC-H: <b>Total hot run time: 31973 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ed0bea85afb9255428029d24f501c0b2c9b97585, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17777 4070 3989 3989
q2 q3 11001 1396 843 843
q4 4817 490 354 354
q5 10548 2296 2135 2135
q6 381 176 144 144
q7 1000 805 640 640
q8 9580 1778 1686 1686
q9 7023 4989 5035 4989
q10 6472 2270 1942 1942
q11 447 274 249 249
q12 676 440 304 304
q13 18174 3455 2789 2789
q14 266 253 251 251
q15 q16 815 773 715 715
q17 1002 887 987 887
q18 6931 5856 6142 5856
q19 1266 1270 1094 1094
q20 533 407 274 274
q21 6021 2805 2503 2503
q22 479 375 329 329
Total cold run time: 105209 ms
Total hot run time: 31973 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4898 4984 4806 4806
q2 q3 4878 5261 4633 4633
q4 2131 2186 1411 1411
q5 4914 4718 4790 4718
q6 241 183 128 128
q7 1864 1603 1439 1439
q8 2243 2031 1972 1972
q9 7435 7432 7442 7432
q10 4781 4687 4201 4201
q11 545 388 356 356
q12 725 744 524 524
q13 3100 3441 2763 2763
q14 278 276 256 256
q15 q16 687 703 618 618
q17 1320 1248 1261 1248
q18 7358 6807 6627 6627
q19 1087 1092 1114 1092
q20 2229 2226 1934 1934
q21 5290 4613 4472 4472
q22 517 489 427 427
Total cold run time: 56521 ms
Total hot run time: 51057 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](BE) split EncodingInfo defaults into 4 explicit maps [doris]
Posted by "csun5285 (via GitHub)" <gi...@apache.org>.
csun5285 commented on PR #63622:
URL: https://github.com/apache/doris/pull/63622#issuecomment-4533815621
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](BE) split EncodingInfo defaults into 4 explicit maps [doris]
Posted by "csun5285 (via GitHub)" <gi...@apache.org>.
csun5285 commented on code in PR #63622:
URL: https://github.com/apache/doris/pull/63622#discussion_r3303562366
##########
be/src/storage/index/primary_key_index.cpp:
##########
@@ -42,7 +42,7 @@ Status PrimaryKeyIndexBuilder::init() {
options.write_ordinal_index = true;
options.write_value_index = true;
options.data_page_size = config::primary_key_data_page_size;
- options.encoding = segment_v2::EncodingInfo::get_default_encoding(type, {}, true);
+ options.encoding = segment_v2::EncodingInfo::get_value_seek_encoding(type);
Review Comment:
是的，其他的没用到的在 value_seek_map 里面的我都删了
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](BE) split EncodingInfo defaults into 4 explicit maps [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63622:
URL: https://github.com/apache/doris/pull/63622#issuecomment-4552990943
<details>
<summary>TPC-DS: <b>Total hot run time: 171768 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7f8b64511849bf86ccc9c1204995af479deecc95, data reload: false
query5 4305 677 534 534
query6 329 230 204 204
query7 4240 592 326 326
query8 344 255 235 235
query9 8827 4102 4134 4102
query10 458 340 300 300
query11 5804 2564 2273 2273
query12 178 132 129 129
query13 1277 617 437 437
query14 6180 5483 5185 5185
query14_1 4494 4552 4492 4492
query15 221 208 189 189
query16 1037 482 449 449
query17 1132 718 603 603
query18 2723 491 356 356
query19 216 206 163 163
query20 137 135 131 131
query21 222 144 124 124
query22 13704 13638 13362 13362
query23 17496 16664 16200 16200
query23_1 16338 16321 16384 16321
query24 7404 1798 1320 1320
query24_1 1337 1314 1310 1310
query25 541 485 406 406
query26 1325 316 181 181
query27 2694 555 353 353
query28 4426 2011 2048 2011
query29 969 638 509 509
query30 312 241 204 204
query31 1172 1093 950 950
query32 104 77 73 73
query33 540 361 298 298
query34 1206 1169 655 655
query35 787 806 717 717
query36 1369 1400 1269 1269
query37 168 112 99 99
query38 3227 3212 3087 3087
query39 950 937 898 898
query39_1 874 887 880 880
query40 243 156 129 129
query41 72 71 70 70
query42 118 115 115 115
query43 346 361 313 313
query44
query45 219 209 205 205
query46 1085 1232 755 755
query47 2413 2361 2265 2265
query48 420 441 310 310
query49 650 519 415 415
query50 1008 373 270 270
query51 4413 4292 4251 4251
query52 108 113 100 100
query53 262 286 223 223
query54 331 290 276 276
query55 100 93 88 88
query56 316 332 335 332
query57 1441 1431 1352 1352
query58 324 296 284 284
query59 1663 1696 1470 1470
query60 345 346 331 331
query61 185 184 184 184
query62 705 648 598 598
query63 260 206 211 206
query64 2402 871 711 711
query65
query66 1675 505 377 377
query67 29721 29764 29501 29501
query68
query69 469 361 317 317
query70 1074 997 989 989
query71 318 295 279 279
query72 3183 2669 2436 2436
query73 885 790 432 432
query74 5151 4925 4823 4823
query75 2700 2618 2278 2278
query76 2263 1186 796 796
query77 420 440 340 340
query78 12485 12483 11852 11852
query79 1513 1101 768 768
query80 1221 548 457 457
query81 507 281 238 238
query82 1351 160 126 126
query83 370 285 256 256
query84 261 146 114 114
query85 924 549 453 453
query86 443 335 330 330
query87 3463 3396 3269 3269
query88 3731 2767 2754 2754
query89 456 409 351 351
query90 1800 188 191 188
query91 204 167 140 140
query92 76 84 75 75
query93 1541 1508 867 867
query94 672 350 307 307
query95 691 493 354 354
query96 1025 827 336 336
query97 2738 2739 2616 2616
query98 234 231 227 227
query99 1125 1167 1023 1023
Total cold run time: 256012 ms
Total hot run time: 171768 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](BE) split EncodingInfo defaults into 4 explicit maps [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63622:
URL: https://github.com/apache/doris/pull/63622#issuecomment-4553480595
# BE UT Coverage Report
Increment line coverage `91.72% (277/302)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/7f8b64511849bf86ccc9c1204995af479deecc95_7f8b64511849bf86ccc9c1204995af479deecc95/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/7f8b64511849bf86ccc9c1204995af479deecc95_7f8b64511849bf86ccc9c1204995af479deecc95/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 53.88% (20922/38829) |
| Line Coverage | 37.41% (197959/529203) |
| Region Coverage | 33.65% (154932/460402) |
| Branch Coverage | 34.67% (67450/194574) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](BE) split EncodingInfo defaults into 4 explicit maps [doris]
Posted by "csun5285 (via GitHub)" <gi...@apache.org>.
csun5285 commented on PR #63622:
URL: https://github.com/apache/doris/pull/63622#issuecomment-4561362479
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](BE) split EncodingInfo defaults into 4 explicit maps [doris]
Posted by "csun5285 (via GitHub)" <gi...@apache.org>.
csun5285 commented on PR #63622:
URL: https://github.com/apache/doris/pull/63622#issuecomment-4560634343
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](BE) split EncodingInfo defaults into 4 explicit maps [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63622:
URL: https://github.com/apache/doris/pull/63622#issuecomment-4562603974
# BE Regression && UT Coverage Report
Increment line coverage `91.32% (284/311)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63622_8eff36afaba543f4d0dca6550f163b964dd15be9_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63622_8eff36afaba543f4d0dca6550f163b964dd15be9_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 72.06% (27424/38055) |
| Line Coverage | 55.39% (292711/528424) |
| Region Coverage | 52.30% (243372/465306) |
| Branch Coverage | 53.64% (104900/195576) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](BE) split EncodingInfo defaults into 4 explicit maps [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63622:
URL: https://github.com/apache/doris/pull/63622#issuecomment-4535320812
<details>
<summary>TPC-H: <b>Total hot run time: 31498 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ebb96fc4a6ab398cc891b04c4acaa659e3a0f741, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17675 4108 4052 4052
q2 q3 10780 1402 822 822
q4 4684 482 341 341
q5 7559 2248 2128 2128
q6 312 176 141 141
q7 937 790 631 631
q8 9391 1754 1534 1534
q9 6822 4982 4954 4954
q10 6455 2241 1937 1937
q11 434 269 239 239
q12 689 429 299 299
q13 18190 3361 2780 2780
q14 266 259 233 233
q15 q16 831 794 709 709
q17 994 952 960 952
q18 6757 5721 5528 5528
q19 1179 1359 1084 1084
q20 530 453 272 272
q21 6163 2773 2540 2540
q22 450 376 322 322
Total cold run time: 101098 ms
Total hot run time: 31498 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4845 4848 4946 4848
q2 q3 4882 5263 4851 4851
q4 2125 2222 1424 1424
q5 4996 4793 4670 4670
q6 228 183 141 141
q7 1871 1751 1565 1565
q8 2429 1986 1956 1956
q9 7548 7541 7430 7430
q10 4790 4722 4256 4256
q11 549 417 361 361
q12 736 743 544 544
q13 2966 3400 2795 2795
q14 279 285 252 252
q15 q16 685 710 626 626
q17 1295 1283 1277 1277
q18 7239 6834 6931 6834
q19 1160 1124 1085 1085
q20 2230 2237 1968 1968
q21 5327 4644 4472 4472
q22 527 462 408 408
Total cold run time: 56707 ms
Total hot run time: 51763 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](BE) split EncodingInfo defaults into 4 explicit maps [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #63622:
URL: https://github.com/apache/doris/pull/63622#discussion_r3308236899
##########
be/src/storage/segment/encoding_info.cpp:
##########
@@ -214,124 +216,181 @@ struct TypeEncodingTraits<type, PREFIX_ENCODING, Slice> {
};
EncodingInfoResolver::EncodingInfoResolver() {
- _add_map<FieldType::OLAP_FIELD_TYPE_TINYINT, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_TINYINT, FOR_ENCODING, true>();
- _add_map<FieldType::OLAP_FIELD_TYPE_TINYINT, PLAIN_ENCODING>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_SMALLINT, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_SMALLINT, FOR_ENCODING, true>();
- _add_map<FieldType::OLAP_FIELD_TYPE_SMALLINT, PLAIN_ENCODING>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_INT, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_INT, FOR_ENCODING, true>();
- _add_map<FieldType::OLAP_FIELD_TYPE_INT, PLAIN_ENCODING>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_BIGINT, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_BIGINT, FOR_ENCODING, true>();
- _add_map<FieldType::OLAP_FIELD_TYPE_BIGINT, PLAIN_ENCODING>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_UNSIGNED_BIGINT, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_UNSIGNED_INT, BIT_SHUFFLE>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_LARGEINT, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_LARGEINT, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_LARGEINT, FOR_ENCODING, true>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_FLOAT, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_FLOAT, PLAIN_ENCODING>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_DOUBLE, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_DOUBLE, PLAIN_ENCODING>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_CHAR, DICT_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_CHAR, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_CHAR, PLAIN_ENCODING_V2>();
- _add_map<FieldType::OLAP_FIELD_TYPE_CHAR, PREFIX_ENCODING, true>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_VARCHAR, DICT_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_VARCHAR, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_VARCHAR, PLAIN_ENCODING_V2>();
- _add_map<FieldType::OLAP_FIELD_TYPE_VARCHAR, PREFIX_ENCODING, true>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_STRING, DICT_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_STRING, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_STRING, PLAIN_ENCODING_V2>();
- _add_map<FieldType::OLAP_FIELD_TYPE_STRING, PREFIX_ENCODING, true>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_JSONB, DICT_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_JSONB, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_JSONB, PLAIN_ENCODING_V2>();
- _add_map<FieldType::OLAP_FIELD_TYPE_JSONB, PREFIX_ENCODING, true>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_VARIANT, DICT_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_VARIANT, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_VARIANT, PLAIN_ENCODING_V2>();
- _add_map<FieldType::OLAP_FIELD_TYPE_VARIANT, PREFIX_ENCODING, true>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_BOOL, RLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_BOOL, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_BOOL, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_BOOL, PLAIN_ENCODING, true>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_DATE, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_DATE, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_DATE, FOR_ENCODING, true>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_DATEV2, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_DATEV2, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_DATEV2, FOR_ENCODING, true>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_DATETIMEV2, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_DATETIMEV2, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_DATETIMEV2, FOR_ENCODING, true>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_DATETIME, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_DATETIME, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_DATETIME, FOR_ENCODING, true>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_TIMESTAMPTZ, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_TIMESTAMPTZ, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_TIMESTAMPTZ, FOR_ENCODING, true>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_DECIMAL, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_DECIMAL, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_DECIMAL, BIT_SHUFFLE, true>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_DECIMAL32, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_DECIMAL32, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_DECIMAL32, BIT_SHUFFLE, true>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_DECIMAL64, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_DECIMAL64, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_DECIMAL64, BIT_SHUFFLE, true>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_DECIMAL128I, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_DECIMAL128I, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_DECIMAL128I, BIT_SHUFFLE, true>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_DECIMAL256, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_DECIMAL256, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_DECIMAL256, BIT_SHUFFLE, true>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_IPV4, BIT_SHUFFLE>();
Review Comment:
ipv4 还有必要bitshuffle 吗
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](BE) split EncodingInfo defaults into 4 explicit maps [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #63622:
URL: https://github.com/apache/doris/pull/63622#discussion_r3303360218
##########
be/src/storage/index/primary_key_index.cpp:
##########
@@ -42,7 +42,7 @@ Status PrimaryKeyIndexBuilder::init() {
options.write_ordinal_index = true;
options.write_value_index = true;
options.data_page_size = config::primary_key_data_page_size;
- options.encoding = segment_v2::EncodingInfo::get_default_encoding(type, {}, true);
+ options.encoding = segment_v2::EncodingInfo::get_value_seek_encoding(type);
Review Comment:
这个type 一定是varchar 吗？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](BE) split EncodingInfo defaults into 4 explicit maps [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #63622:
URL: https://github.com/apache/doris/pull/63622#issuecomment-4553897576
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](BE) split EncodingInfo defaults into 4 explicit maps [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63622:
URL: https://github.com/apache/doris/pull/63622#issuecomment-4550645358
<details>
<summary>TPC-H: <b>Total hot run time: 32014 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b5923d7c486a43fa7db4d2f3e0845c93f01b59f3, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17768 4102 4084 4084
q2 q3 10787 1429 841 841
q4 4721 481 346 346
q5 7987 2324 2131 2131
q6 359 185 143 143
q7 964 799 653 653
q8 9375 1694 1726 1694
q9 7137 4993 4983 4983
q10 6474 2306 1887 1887
q11 447 276 255 255
q12 695 438 300 300
q13 18194 3509 2818 2818
q14 268 265 239 239
q15 q16 819 774 723 723
q17 897 961 1005 961
q18 7104 5844 5550 5550
q19 1286 1352 1167 1167
q20 533 408 277 277
q21 5898 2815 2644 2644
q22 471 376 318 318
Total cold run time: 102184 ms
Total hot run time: 32014 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4857 4984 4795 4795
q2 q3 4911 5407 4687 4687
q4 2186 2237 1481 1481
q5 5027 4704 4795 4704
q6 246 196 142 142
q7 1925 1794 1607 1607
q8 2478 1973 2022 1973
q9 7528 7509 7490 7490
q10 4840 4731 4242 4242
q11 564 422 361 361
q12 746 757 566 566
q13 3021 3476 2821 2821
q14 281 292 259 259
q15 q16 699 703 628 628
q17 1329 1314 1297 1297
q18 7563 6789 6853 6789
q19 1207 1106 1155 1106
q20 2261 2256 1962 1962
q21 5437 4746 4584 4584
q22 555 493 425 425
Total cold run time: 57661 ms
Total hot run time: 51919 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](BE) split EncodingInfo defaults into 4 explicit maps [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #63622:
URL: https://github.com/apache/doris/pull/63622#discussion_r3308224868
##########
be/src/storage/segment/encoding_info.cpp:
##########
@@ -214,124 +216,181 @@ struct TypeEncodingTraits<type, PREFIX_ENCODING, Slice> {
};
EncodingInfoResolver::EncodingInfoResolver() {
- _add_map<FieldType::OLAP_FIELD_TYPE_TINYINT, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_TINYINT, FOR_ENCODING, true>();
- _add_map<FieldType::OLAP_FIELD_TYPE_TINYINT, PLAIN_ENCODING>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_SMALLINT, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_SMALLINT, FOR_ENCODING, true>();
- _add_map<FieldType::OLAP_FIELD_TYPE_SMALLINT, PLAIN_ENCODING>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_INT, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_INT, FOR_ENCODING, true>();
- _add_map<FieldType::OLAP_FIELD_TYPE_INT, PLAIN_ENCODING>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_BIGINT, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_BIGINT, FOR_ENCODING, true>();
- _add_map<FieldType::OLAP_FIELD_TYPE_BIGINT, PLAIN_ENCODING>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_UNSIGNED_BIGINT, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_UNSIGNED_INT, BIT_SHUFFLE>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_LARGEINT, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_LARGEINT, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_LARGEINT, FOR_ENCODING, true>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_FLOAT, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_FLOAT, PLAIN_ENCODING>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_DOUBLE, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_DOUBLE, PLAIN_ENCODING>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_CHAR, DICT_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_CHAR, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_CHAR, PLAIN_ENCODING_V2>();
- _add_map<FieldType::OLAP_FIELD_TYPE_CHAR, PREFIX_ENCODING, true>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_VARCHAR, DICT_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_VARCHAR, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_VARCHAR, PLAIN_ENCODING_V2>();
- _add_map<FieldType::OLAP_FIELD_TYPE_VARCHAR, PREFIX_ENCODING, true>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_STRING, DICT_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_STRING, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_STRING, PLAIN_ENCODING_V2>();
- _add_map<FieldType::OLAP_FIELD_TYPE_STRING, PREFIX_ENCODING, true>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_JSONB, DICT_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_JSONB, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_JSONB, PLAIN_ENCODING_V2>();
- _add_map<FieldType::OLAP_FIELD_TYPE_JSONB, PREFIX_ENCODING, true>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_VARIANT, DICT_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_VARIANT, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_VARIANT, PLAIN_ENCODING_V2>();
- _add_map<FieldType::OLAP_FIELD_TYPE_VARIANT, PREFIX_ENCODING, true>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_BOOL, RLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_BOOL, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_BOOL, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_BOOL, PLAIN_ENCODING, true>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_DATE, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_DATE, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_DATE, FOR_ENCODING, true>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_DATEV2, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_DATEV2, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_DATEV2, FOR_ENCODING, true>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_DATETIMEV2, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_DATETIMEV2, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_DATETIMEV2, FOR_ENCODING, true>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_DATETIME, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_DATETIME, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_DATETIME, FOR_ENCODING, true>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_TIMESTAMPTZ, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_TIMESTAMPTZ, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_TIMESTAMPTZ, FOR_ENCODING, true>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_DECIMAL, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_DECIMAL, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_DECIMAL, BIT_SHUFFLE, true>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_DECIMAL32, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_DECIMAL32, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_DECIMAL32, BIT_SHUFFLE, true>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_DECIMAL64, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_DECIMAL64, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_DECIMAL64, BIT_SHUFFLE, true>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_DECIMAL128I, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_DECIMAL128I, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_DECIMAL128I, BIT_SHUFFLE, true>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_DECIMAL256, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_DECIMAL256, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_DECIMAL256, BIT_SHUFFLE, true>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_IPV4, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_IPV4, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_IPV4, BIT_SHUFFLE, true>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_IPV6, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_IPV6, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_IPV6, BIT_SHUFFLE, true>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_HLL, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_HLL, PLAIN_ENCODING_V2>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_BITMAP, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_BITMAP, PLAIN_ENCODING_V2>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_QUANTILE_STATE, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_QUANTILE_STATE, PLAIN_ENCODING_V2>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_AGG_STATE, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_AGG_STATE, PLAIN_ENCODING_V2>();
+ // ===== Phase 1: register every supported (type, encoding) combination exactly once =====
+ // _register_supported_encoding CHECKs against duplicates; the Phase 2 calls below do not insert into _encoding_map,
+ // so every (type, encoding) used as a default must appear here first.
+
+ // signed integers
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_TINYINT, BIT_SHUFFLE>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_TINYINT, PLAIN_ENCODING>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_TINYINT, FOR_ENCODING>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_SMALLINT, BIT_SHUFFLE>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_SMALLINT, PLAIN_ENCODING>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_SMALLINT, FOR_ENCODING>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_INT, BIT_SHUFFLE>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_INT, PLAIN_ENCODING>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_INT, FOR_ENCODING>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_BIGINT, BIT_SHUFFLE>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_BIGINT, PLAIN_ENCODING>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_BIGINT, FOR_ENCODING>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_LARGEINT, BIT_SHUFFLE>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_LARGEINT, PLAIN_ENCODING>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_LARGEINT, FOR_ENCODING>();
+
+ // unsigned integers
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_UNSIGNED_BIGINT, BIT_SHUFFLE>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_UNSIGNED_INT, BIT_SHUFFLE>();
+
+ // FLOAT / DOUBLE
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_FLOAT, BIT_SHUFFLE>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_FLOAT, PLAIN_ENCODING>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_DOUBLE, BIT_SHUFFLE>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_DOUBLE, PLAIN_ENCODING>();
+
+ // binary types (CHAR/VARCHAR/STRING/JSONB/VARIANT)
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_CHAR, DICT_ENCODING>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_CHAR, PLAIN_ENCODING>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_CHAR, PLAIN_ENCODING_V2>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_CHAR, PREFIX_ENCODING>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_VARCHAR, DICT_ENCODING>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_VARCHAR, PLAIN_ENCODING>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_VARCHAR, PLAIN_ENCODING_V2>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_VARCHAR, PREFIX_ENCODING>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_STRING, DICT_ENCODING>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_STRING, PLAIN_ENCODING>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_STRING, PLAIN_ENCODING_V2>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_STRING, PREFIX_ENCODING>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_JSONB, DICT_ENCODING>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_JSONB, PLAIN_ENCODING>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_JSONB, PLAIN_ENCODING_V2>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_JSONB, PREFIX_ENCODING>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_VARIANT, DICT_ENCODING>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_VARIANT, PLAIN_ENCODING>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_VARIANT, PLAIN_ENCODING_V2>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_VARIANT, PREFIX_ENCODING>();
+
+ // BOOL
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_BOOL, RLE>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_BOOL, BIT_SHUFFLE>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_BOOL, PLAIN_ENCODING>();
+
+ // date / datetime
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_DATE, BIT_SHUFFLE>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_DATE, PLAIN_ENCODING>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_DATE, FOR_ENCODING>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_DATEV2, BIT_SHUFFLE>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_DATEV2, PLAIN_ENCODING>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_DATEV2, FOR_ENCODING>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_DATETIMEV2, BIT_SHUFFLE>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_DATETIMEV2, PLAIN_ENCODING>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_DATETIMEV2, FOR_ENCODING>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_DATETIME, BIT_SHUFFLE>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_DATETIME, PLAIN_ENCODING>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_DATETIME, FOR_ENCODING>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_TIMESTAMPTZ, BIT_SHUFFLE>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_TIMESTAMPTZ, PLAIN_ENCODING>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_TIMESTAMPTZ, FOR_ENCODING>();
+
+ // decimal
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_DECIMAL, BIT_SHUFFLE>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_DECIMAL, PLAIN_ENCODING>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_DECIMAL32, BIT_SHUFFLE>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_DECIMAL32, PLAIN_ENCODING>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_DECIMAL64, BIT_SHUFFLE>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_DECIMAL64, PLAIN_ENCODING>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_DECIMAL128I, BIT_SHUFFLE>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_DECIMAL128I, PLAIN_ENCODING>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_DECIMAL256, BIT_SHUFFLE>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_DECIMAL256, PLAIN_ENCODING>();
+
+ // ip
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_IPV4, BIT_SHUFFLE>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_IPV4, PLAIN_ENCODING>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_IPV6, BIT_SHUFFLE>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_IPV6, PLAIN_ENCODING>();
+
+ // aggregate / binary-flavored types
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_HLL, PLAIN_ENCODING>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_HLL, PLAIN_ENCODING_V2>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_BITMAP, PLAIN_ENCODING>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_BITMAP, PLAIN_ENCODING_V2>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_QUANTILE_STATE, PLAIN_ENCODING>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_QUANTILE_STATE, PLAIN_ENCODING_V2>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_AGG_STATE, PLAIN_ENCODING>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_AGG_STATE, PLAIN_ENCODING_V2>();
+
+ // ===== Phase 2a: V2 defaults (write path, V1/V2 segments) =====
+ _set_v2_default<FieldType::OLAP_FIELD_TYPE_TINYINT, BIT_SHUFFLE>();
+ _set_v2_default<FieldType::OLAP_FIELD_TYPE_SMALLINT, BIT_SHUFFLE>();
+ _set_v2_default<FieldType::OLAP_FIELD_TYPE_INT, BIT_SHUFFLE>();
+ _set_v2_default<FieldType::OLAP_FIELD_TYPE_BIGINT, BIT_SHUFFLE>();
+ _set_v2_default<FieldType::OLAP_FIELD_TYPE_LARGEINT, BIT_SHUFFLE>();
+ _set_v2_default<FieldType::OLAP_FIELD_TYPE_UNSIGNED_BIGINT, BIT_SHUFFLE>();
+ _set_v2_default<FieldType::OLAP_FIELD_TYPE_UNSIGNED_INT, BIT_SHUFFLE>();
+ _set_v2_default<FieldType::OLAP_FIELD_TYPE_FLOAT, BIT_SHUFFLE>();
+ _set_v2_default<FieldType::OLAP_FIELD_TYPE_DOUBLE, BIT_SHUFFLE>();
+ _set_v2_default<FieldType::OLAP_FIELD_TYPE_CHAR, DICT_ENCODING>();
+ _set_v2_default<FieldType::OLAP_FIELD_TYPE_VARCHAR, DICT_ENCODING>();
+ _set_v2_default<FieldType::OLAP_FIELD_TYPE_STRING, DICT_ENCODING>();
+ _set_v2_default<FieldType::OLAP_FIELD_TYPE_JSONB, DICT_ENCODING>();
+ _set_v2_default<FieldType::OLAP_FIELD_TYPE_VARIANT, DICT_ENCODING>();
+ _set_v2_default<FieldType::OLAP_FIELD_TYPE_BOOL, RLE>();
+ _set_v2_default<FieldType::OLAP_FIELD_TYPE_DATE, BIT_SHUFFLE>();
+ _set_v2_default<FieldType::OLAP_FIELD_TYPE_DATEV2, BIT_SHUFFLE>();
+ _set_v2_default<FieldType::OLAP_FIELD_TYPE_DATETIMEV2, BIT_SHUFFLE>();
+ _set_v2_default<FieldType::OLAP_FIELD_TYPE_DATETIME, BIT_SHUFFLE>();
+ _set_v2_default<FieldType::OLAP_FIELD_TYPE_TIMESTAMPTZ, BIT_SHUFFLE>();
+ _set_v2_default<FieldType::OLAP_FIELD_TYPE_DECIMAL, BIT_SHUFFLE>();
+ _set_v2_default<FieldType::OLAP_FIELD_TYPE_DECIMAL32, BIT_SHUFFLE>();
+ _set_v2_default<FieldType::OLAP_FIELD_TYPE_DECIMAL64, BIT_SHUFFLE>();
+ _set_v2_default<FieldType::OLAP_FIELD_TYPE_DECIMAL128I, BIT_SHUFFLE>();
+ _set_v2_default<FieldType::OLAP_FIELD_TYPE_DECIMAL256, BIT_SHUFFLE>();
+ _set_v2_default<FieldType::OLAP_FIELD_TYPE_IPV4, BIT_SHUFFLE>();
+ _set_v2_default<FieldType::OLAP_FIELD_TYPE_IPV6, BIT_SHUFFLE>();
+ _set_v2_default<FieldType::OLAP_FIELD_TYPE_HLL, PLAIN_ENCODING>();
+ _set_v2_default<FieldType::OLAP_FIELD_TYPE_BITMAP, PLAIN_ENCODING>();
+ _set_v2_default<FieldType::OLAP_FIELD_TYPE_QUANTILE_STATE, PLAIN_ENCODING>();
+ _set_v2_default<FieldType::OLAP_FIELD_TYPE_AGG_STATE, PLAIN_ENCODING>();
+
+ // ===== Phase 2b: V3 defaults (write path, V3 segments) =====
+ _set_v3_default<FieldType::OLAP_FIELD_TYPE_TINYINT, PLAIN_ENCODING>();
+ _set_v3_default<FieldType::OLAP_FIELD_TYPE_SMALLINT, PLAIN_ENCODING>();
+ _set_v3_default<FieldType::OLAP_FIELD_TYPE_INT, PLAIN_ENCODING>();
+ _set_v3_default<FieldType::OLAP_FIELD_TYPE_BIGINT, PLAIN_ENCODING>();
+ _set_v3_default<FieldType::OLAP_FIELD_TYPE_LARGEINT, PLAIN_ENCODING>();
+ _set_v3_default<FieldType::OLAP_FIELD_TYPE_UNSIGNED_BIGINT, BIT_SHUFFLE>();
Review Comment:
这4个数值类型，确定是bit shuffle 吗？ 感觉需要跟包含确认一下，另外，确认一下clickhouse的代码
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](BE) split EncodingInfo defaults into 4 explicit maps [doris]
Posted by "csun5285 (via GitHub)" <gi...@apache.org>.
csun5285 commented on code in PR #63622:
URL: https://github.com/apache/doris/pull/63622#discussion_r3301579543
##########
be/src/storage/segment/vertical_segment_writer.cpp:
##########
@@ -160,11 +161,14 @@ VerticalSegmentWriter::~VerticalSegmentWriter() {
}
void VerticalSegmentWriter::_init_column_meta(ColumnMetaPB* meta, uint32_t column_id,
Review Comment:
传到下面 column writer 也要用，比如struct 的里面列创建的时候使用 V3 的编码
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](BE) split EncodingInfo defaults into 4 explicit maps [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63622:
URL: https://github.com/apache/doris/pull/63622#issuecomment-4561599479
<details>
<summary>TPC-DS: <b>Total hot run time: 171787 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ed0bea85afb9255428029d24f501c0b2c9b97585, data reload: false
query5 4313 676 534 534
query6 330 214 199 199
query7 4226 596 330 330
query8 326 237 217 217
query9 8833 4119 4082 4082
query10 444 334 305 305
query11 5779 2497 2287 2287
query12 196 131 130 130
query13 1291 620 443 443
query14 6181 5505 5198 5198
query14_1 4503 4515 4496 4496
query15 219 213 188 188
query16 1039 464 464 464
query17 1168 751 606 606
query18 2506 504 382 382
query19 217 209 171 171
query20 140 133 128 128
query21 222 146 118 118
query22 13715 13612 13420 13420
query23 17423 16536 16249 16249
query23_1 16386 16399 16172 16172
query24 7464 1775 1339 1339
query24_1 1341 1306 1314 1306
query25 547 473 414 414
query26 1320 327 172 172
query27 2690 565 345 345
query28 4407 2032 2038 2032
query29 976 620 492 492
query30 304 241 200 200
query31 1134 1079 939 939
query32 102 77 73 73
query33 546 373 316 316
query34 1197 1152 658 658
query35 785 794 715 715
query36 1400 1366 1324 1324
query37 153 106 89 89
query38 3201 3153 3082 3082
query39 949 921 922 921
query39_1 888 889 877 877
query40 222 147 123 123
query41 65 64 62 62
query42 113 107 112 107
query43 337 342 300 300
query44
query45 220 207 194 194
query46 1067 1256 738 738
query47 2411 2409 2223 2223
query48 403 426 320 320
query49 632 497 390 390
query50 953 345 269 269
query51 4316 4405 4262 4262
query52 105 106 94 94
query53 259 274 208 208
query54 317 269 267 267
query55 98 95 88 88
query56 314 305 307 305
query57 1435 1415 1310 1310
query58 305 270 268 268
query59 1587 1701 1439 1439
query60 310 332 301 301
query61 160 159 158 158
query62 700 652 590 590
query63 242 199 206 199
query64 2442 800 632 632
query65
query66 1759 493 361 361
query67 29759 29701 29521 29521
query68
query69 469 345 322 322
query70 1054 1011 1063 1011
query71 312 276 265 265
query72 3027 2668 2453 2453
query73 862 807 438 438
query74 5133 4955 4793 4793
query75 2742 2611 2274 2274
query76 2282 1153 762 762
query77 406 424 343 343
query78 12597 12469 11877 11877
query79 1270 1082 737 737
query80 578 536 458 458
query81 448 280 240 240
query82 244 172 122 122
query83 280 283 251 251
query84 252 147 114 114
query85 853 550 452 452
query86 376 371 313 313
query87 3462 3344 3291 3291
query88 3641 2776 2748 2748
query89 434 397 347 347
query90 2229 202 194 194
query91 216 176 146 146
query92 83 86 78 78
query93 1556 1586 879 879
query94 550 376 334 334
query95 685 488 354 354
query96 1047 832 372 372
query97 2708 2734 2617 2617
query98 244 236 251 236
query99 1153 1137 1026 1026
Total cold run time: 253617 ms
Total hot run time: 171787 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](BE) split EncodingInfo defaults into 4 explicit maps [doris]
Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #63622:
URL: https://github.com/apache/doris/pull/63622#issuecomment-4560641653
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](BE) split EncodingInfo defaults into 4 explicit maps [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on PR #63622:
URL: https://github.com/apache/doris/pull/63622#issuecomment-4551900030
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](BE) split EncodingInfo defaults into 4 explicit maps [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #63622:
URL: https://github.com/apache/doris/pull/63622#discussion_r3302168081
##########
be/src/storage/segment/encoding_info.h:
##########
@@ -51,15 +51,19 @@ class DataPagePreDecoder {
class EncodingInfo {
public:
- // Get EncodingInfo for TypeInfo and EncodingTypePB
- static Status get(FieldType type, EncodingTypePB encoding_type,
- EncodingPreference encoding_preference, const EncodingInfo** encoding);
+ // Look up the EncodingInfo for an already-resolved (type, encoding) pair.
+ // Read paths use this directly; write paths first resolve a default via the
+ // get_*_default_encoding helpers below.
+ static Status get(FieldType type, EncodingTypePB encoding_type, const EncodingInfo** encoding);
- // optimize_value_search: whether the encoding scheme should optimize for ordered data
- // and support fast value seek operation
- static EncodingTypePB get_default_encoding(FieldType type,
- EncodingPreference encoding_preference,
- bool optimize_value_seek);
+ // Default column encoding for the legacy (V1/V2 segment) write path.
+ static EncodingTypePB get_legacy_default_encoding(FieldType type);
Review Comment:
rename to get_v2_default_encoding
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](BE) split EncodingInfo defaults into 4 explicit maps [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #63622:
URL: https://github.com/apache/doris/pull/63622#issuecomment-4553881768
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](BE) split EncodingInfo defaults into 4 explicit maps [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63622:
URL: https://github.com/apache/doris/pull/63622#issuecomment-4552911619
<details>
<summary>TPC-H: <b>Total hot run time: 31874 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7f8b64511849bf86ccc9c1204995af479deecc95, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17645 4089 4134 4089
q2 q3 10817 1421 811 811
q4 4686 477 344 344
q5 7559 2268 2099 2099
q6 242 179 142 142
q7 973 780 636 636
q8 9397 1859 1632 1632
q9 5129 4979 4991 4979
q10 6405 2193 1937 1937
q11 444 277 250 250
q12 632 423 309 309
q13 18091 3392 2734 2734
q14 283 260 241 241
q15 q16 817 787 712 712
q17 892 973 898 898
q18 6996 5632 5525 5525
q19 1202 1325 1136 1136
q20 649 452 297 297
q21 5407 2807 2724 2724
q22 456 379 455 379
Total cold run time: 98722 ms
Total hot run time: 31874 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4786 4880 4864 4864
q2 q3 4924 5349 4613 4613
q4 2140 2190 1404 1404
q5 4784 4949 4791 4791
q6 234 178 125 125
q7 1871 1827 1610 1610
q8 2440 2204 2246 2204
q9 7913 7556 7466 7466
q10 4746 4688 4200 4200
q11 536 402 355 355
q12 734 742 532 532
q13 3150 3508 2846 2846
q14 265 276 255 255
q15 q16 694 702 627 627
q17 1300 1263 1270 1263
q18 7359 6946 6925 6925
q19 1206 1103 1087 1087
q20 2242 2220 1976 1976
q21 5327 4685 4530 4530
q22 533 472 407 407
Total cold run time: 57184 ms
Total hot run time: 52080 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](BE) split EncodingInfo defaults into 4 explicit maps [doris]
Posted by "csun5285 (via GitHub)" <gi...@apache.org>.
csun5285 commented on PR #63622:
URL: https://github.com/apache/doris/pull/63622#issuecomment-4550364740
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](BE) split EncodingInfo defaults into 4 explicit maps [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei merged PR #63622:
URL: https://github.com/apache/doris/pull/63622
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](BE) split EncodingInfo defaults into 4 explicit maps [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63622:
URL: https://github.com/apache/doris/pull/63622#issuecomment-4562603141
# FE Regression Coverage Report
Increment line coverage ` 3.17% (2/63)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63622_8eff36afaba543f4d0dca6550f163b964dd15be9_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63622_8eff36afaba543f4d0dca6550f163b964dd15be9_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](BE) split EncodingInfo defaults into 4 explicit maps [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63622:
URL: https://github.com/apache/doris/pull/63622#issuecomment-4550693325
<details>
<summary>TPC-DS: <b>Total hot run time: 173506 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b5923d7c486a43fa7db4d2f3e0845c93f01b59f3, data reload: false
query5 4327 669 518 518
query6 352 213 199 199
query7 4287 553 310 310
query8 325 237 217 217
query9 8852 4218 4160 4160
query10 479 339 301 301
query11 5802 2562 2219 2219
query12 190 129 127 127
query13 1264 613 432 432
query14 6261 5549 5256 5256
query14_1 4577 4576 4556 4556
query15 224 210 190 190
query16 1017 504 473 473
query17 1169 774 629 629
query18 2520 496 376 376
query19 227 212 179 179
query20 145 136 164 136
query21 216 138 121 121
query22 13731 13599 13368 13368
query23 17556 16731 16344 16344
query23_1 16351 16340 16396 16340
query24 7501 1777 1320 1320
query24_1 1326 1339 1337 1337
query25 564 464 412 412
query26 1323 333 172 172
query27 2669 564 349 349
query28 4442 2033 2083 2033
query29 983 638 503 503
query30 321 249 206 206
query31 1146 1075 948 948
query32 88 77 76 76
query33 546 367 303 303
query34 1192 1179 664 664
query35 783 795 712 712
query36 1428 1423 1220 1220
query37 156 107 93 93
query38 3216 3184 3004 3004
query39 939 960 902 902
query39_1 897 859 904 859
query40 243 149 132 132
query41 67 63 63 63
query42 111 113 111 111
query43 338 339 296 296
query44
query45 212 204 201 201
query46 1085 1254 720 720
query47 2373 2379 2261 2261
query48 388 415 298 298
query49 643 503 389 389
query50 1007 356 252 252
query51 4374 4320 4372 4320
query52 106 105 95 95
query53 258 293 207 207
query54 311 277 257 257
query55 96 91 89 89
query56 305 312 332 312
query57 1453 1451 1344 1344
query58 315 289 283 283
query59 1628 1752 1521 1521
query60 335 328 325 325
query61 182 179 180 179
query62 713 658 594 594
query63 250 206 214 206
query64 2483 857 703 703
query65
query66 1756 495 376 376
query67 29869 29860 29762 29762
query68
query69 482 350 322 322
query70 1051 1012 1027 1012
query71 313 276 316 276
query72 3112 2721 2429 2429
query73 835 760 453 453
query74 5153 4968 4843 4843
query75 2698 2630 2293 2293
query76 2313 1183 776 776
query77 411 419 342 342
query78 12651 12575 12033 12033
query79 1512 1079 754 754
query80 1273 548 466 466
query81 511 302 261 261
query82 1362 162 123 123
query83 361 277 252 252
query84 258 141 116 116
query85 952 536 462 462
query86 463 361 351 351
query87 3473 3394 3250 3250
query88 3663 2790 2784 2784
query89 462 401 342 342
query90 1806 188 183 183
query91 176 166 138 138
query92 82 77 75 75
query93 1471 1385 908 908
query94 670 363 311 311
query95 695 465 349 349
query96 1045 794 343 343
query97 2821 2746 2608 2608
query98 242 228 227 227
query99 1216 1165 1036 1036
Total cold run time: 256854 ms
Total hot run time: 173506 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](BE) split EncodingInfo defaults into 4 explicit maps [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63622:
URL: https://github.com/apache/doris/pull/63622#issuecomment-4561379446
# BE UT Coverage Report
Increment line coverage `91.75% (278/303)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/ed0bea85afb9255428029d24f501c0b2c9b97585_ed0bea85afb9255428029d24f501c0b2c9b97585/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/ed0bea85afb9255428029d24f501c0b2c9b97585_ed0bea85afb9255428029d24f501c0b2c9b97585/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 53.96% (20965/38856) |
| Line Coverage | 37.50% (198645/529760) |
| Region Coverage | 33.79% (155719/460841) |
| Branch Coverage | 34.77% (67745/194830) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](BE) split EncodingInfo defaults into 4 explicit maps [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63622:
URL: https://github.com/apache/doris/pull/63622#issuecomment-4562493165
<details>
<summary>TPC-DS: <b>Total hot run time: 172187 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8eff36afaba543f4d0dca6550f163b964dd15be9, data reload: false
query5 4344 658 514 514
query6 331 218 199 199
query7 4225 563 298 298
query8 323 248 221 221
query9 8788 4024 4037 4024
query10 448 335 298 298
query11 5789 2481 2308 2308
query12 177 131 127 127
query13 1266 621 442 442
query14 6132 5523 5162 5162
query14_1 4505 4529 4488 4488
query15 224 211 189 189
query16 1019 475 446 446
query17 1158 776 608 608
query18 2714 490 356 356
query19 217 201 178 178
query20 136 131 131 131
query21 216 138 118 118
query22 13663 13615 13416 13416
query23 17305 16573 16237 16237
query23_1 16359 16275 16281 16275
query24 7265 1765 1319 1319
query24_1 1331 1346 1310 1310
query25 546 475 407 407
query26 1326 330 170 170
query27 2658 595 335 335
query28 4280 2021 2006 2006
query29 996 630 503 503
query30 304 255 202 202
query31 1128 1083 953 953
query32 97 76 74 74
query33 533 343 311 311
query34 1168 1164 651 651
query35 786 795 707 707
query36 1408 1436 1239 1239
query37 156 105 105 105
query38 3238 3166 3100 3100
query39 930 931 903 903
query39_1 867 892 883 883
query40 241 146 125 125
query41 97 62 63 62
query42 111 110 108 108
query43 326 329 290 290
query44
query45 223 213 194 194
query46 1125 1176 772 772
query47 2351 2349 2232 2232
query48 377 409 320 320
query49 631 498 399 399
query50 999 358 259 259
query51 4372 4338 4347 4338
query52 107 104 94 94
query53 259 282 204 204
query54 309 282 255 255
query55 95 87 82 82
query56 320 333 293 293
query57 1429 1413 1343 1343
query58 309 273 270 270
query59 1590 1681 1433 1433
query60 322 334 315 315
query61 161 158 158 158
query62 695 652 589 589
query63 242 201 212 201
query64 2364 800 672 672
query65
query66 1667 480 365 365
query67 29905 29903 29616 29616
query68
query69 469 356 318 318
query70 993 986 931 931
query71 312 279 274 274
query72 3015 2698 2343 2343
query73 859 801 449 449
query74 5105 4941 4792 4792
query75 2685 2626 2267 2267
query76 2105 1170 771 771
query77 401 415 334 334
query78 12490 12369 11892 11892
query79 1284 1019 748 748
query80 582 552 479 479
query81 460 287 249 249
query82 236 162 125 125
query83 282 287 265 265
query84 262 149 117 117
query85 933 618 490 490
query86 359 322 333 322
query87 3419 3379 3253 3253
query88 3632 2765 2738 2738
query89 431 392 345 345
query90 2074 183 185 183
query91 177 171 137 137
query92 82 83 71 71
query93 1423 1382 943 943
query94 542 350 315 315
query95 676 485 348 348
query96 1073 795 345 345
query97 2749 2730 2597 2597
query98 245 227 228 227
query99 1167 1183 1038 1038
Total cold run time: 252484 ms
Total hot run time: 172187 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](BE) split EncodingInfo defaults into 4 explicit maps [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63622:
URL: https://github.com/apache/doris/pull/63622#issuecomment-4536003976
# BE Regression && UT Coverage Report
Increment line coverage `92.74% (281/303)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63622_ebb96fc4a6ab398cc891b04c4acaa659e3a0f741_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63622_ebb96fc4a6ab398cc891b04c4acaa659e3a0f741_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 73.83% (28056/38001) |
| Line Coverage | 57.71% (304513/527640) |
| Region Coverage | 54.83% (254631/464427) |
| Branch Coverage | 56.39% (110048/195169) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](BE) split EncodingInfo defaults into 4 explicit maps [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on PR #63622:
URL: https://github.com/apache/doris/pull/63622#issuecomment-4553507733
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](BE) split EncodingInfo defaults into 4 explicit maps [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63622:
URL: https://github.com/apache/doris/pull/63622#issuecomment-4553841756
# BE Regression && UT Coverage Report
Increment line coverage `91.29% (283/310)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63622_7f8b64511849bf86ccc9c1204995af479deecc95_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63622_7f8b64511849bf86ccc9c1204995af479deecc95_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 73.91% (28107/38029) |
| Line Coverage | 57.80% (305098/527856) |
| Region Coverage | 55.00% (255649/464831) |
| Branch Coverage | 56.54% (110431/195304) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](BE) split EncodingInfo defaults into 4 explicit maps [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63622:
URL: https://github.com/apache/doris/pull/63622#issuecomment-4553888406
# FE Regression Coverage Report
Increment line coverage ` 5.13% (2/39)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63622_7f8b64511849bf86ccc9c1204995af479deecc95_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63622_7f8b64511849bf86ccc9c1204995af479deecc95_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](BE) split EncodingInfo defaults into 4 explicit maps [doris]
Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #63622:
URL: https://github.com/apache/doris/pull/63622#issuecomment-4560513511
Found one correctness issue in the variant writer path:
`prepare_materialized_subcolumn_writer()` creates a fresh `ColumnWriterOptions opts` and copies `meta`, `index_file_writer`, `compression_type`, `rowset_ctx`, and `file_writer`, but it does not copy `base_opts.storage_format`. Since `_init_column_meta()` now resolves the physical encoding from `opts.storage_format`, this path falls back to the `ColumnWriterOptions` default (`TABLET_STORAGE_FORMAT_V2`) even when the owning tablet is V3.
This is different from `prepare_subcolumn_writer_target()`, which now correctly does `opts.storage_format = base_opts.storage_format`. For V3 variant doc materialized subcolumns created via `VariantDocWriter::_write_materialized_subcolumn()` / `VariantDocCompactWriter::_write_materialized_subcolumn()`, the missing assignment can write V2 default encodings and defeat the new storage-format based policy.
Please propagate `opts.storage_format = base_opts.storage_format` in `prepare_materialized_subcolumn_writer()` as well, and consider adding a V3 doc-mode materialized subcolumn test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](BE) split EncodingInfo defaults into 4 explicit maps [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63622:
URL: https://github.com/apache/doris/pull/63622#issuecomment-4535385040
<details>
<summary>TPC-DS: <b>Total hot run time: 173081 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ebb96fc4a6ab398cc891b04c4acaa659e3a0f741, data reload: false
query5 4312 659 527 527
query6 353 225 199 199
query7 4232 553 301 301
query8 324 260 227 227
query9 8844 4192 4145 4145
query10 446 350 306 306
query11 5709 2550 2182 2182
query12 181 130 123 123
query13 1282 606 448 448
query14 6214 5632 5317 5317
query14_1 4682 4681 4659 4659
query15 221 211 189 189
query16 1006 456 437 437
query17 1162 755 616 616
query18 2453 487 367 367
query19 227 214 185 185
query20 138 135 135 135
query21 218 143 120 120
query22 13799 13584 13434 13434
query23 17482 16550 16392 16392
query23_1 16324 16395 16400 16395
query24 7645 1761 1335 1335
query24_1 1337 1324 1321 1321
query25 570 477 426 426
query26 1288 327 170 170
query27 2687 574 363 363
query28 4478 2033 2049 2033
query29 1018 639 489 489
query30 319 249 206 206
query31 1133 1093 959 959
query32 101 77 75 75
query33 539 354 299 299
query34 1186 1146 677 677
query35 770 810 703 703
query36 1388 1406 1275 1275
query37 154 107 92 92
query38 3247 3233 3087 3087
query39 934 920 899 899
query39_1 875 887 894 887
query40 234 146 130 130
query41 67 65 64 64
query42 115 111 111 111
query43 351 352 311 311
query44
query45 223 209 199 199
query46 1131 1212 785 785
query47 2445 2411 2246 2246
query48 419 405 305 305
query49 638 508 425 425
query50 1123 354 249 249
query51 4438 4354 4318 4318
query52 108 108 95 95
query53 268 285 211 211
query54 309 282 282 282
query55 94 95 89 89
query56 307 314 313 313
query57 1481 1449 1400 1400
query58 311 288 272 272
query59 1692 1757 1537 1537
query60 322 329 322 322
query61 163 160 155 155
query62 708 641 585 585
query63 247 211 217 211
query64 2418 806 626 626
query65
query66 2240 483 368 368
query67 29892 29677 29458 29458
query68
query69 480 350 309 309
query70 1063 1012 1040 1012
query71 306 283 270 270
query72 3024 2644 2426 2426
query73 840 745 441 441
query74 5127 4952 4850 4850
query75 2695 2643 2265 2265
query76 2312 1134 783 783
query77 409 429 328 328
query78 12227 12367 11757 11757
query79 1268 1016 752 752
query80 559 544 461 461
query81 449 280 242 242
query82 246 158 125 125
query83 277 284 256 256
query84 267 141 113 113
query85 849 551 449 449
query86 360 368 360 360
query87 3393 3382 3260 3260
query88 3595 2737 2752 2737
query89 431 391 339 339
query90 2167 184 179 179
query91 178 165 138 138
query92 81 80 75 75
query93 1362 1353 917 917
query94 529 353 313 313
query95 685 398 439 398
query96 1074 793 372 372
query97 2741 2748 2592 2592
query98 236 231 230 230
query99 1168 1154 1027 1027
Total cold run time: 254509 ms
Total hot run time: 173081 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](BE) split EncodingInfo defaults into 4 explicit maps [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #63622:
URL: https://github.com/apache/doris/pull/63622#discussion_r3303327606
##########
be/src/storage/segment/encoding_info.cpp:
##########
@@ -214,124 +215,203 @@ struct TypeEncodingTraits<type, PREFIX_ENCODING, Slice> {
};
EncodingInfoResolver::EncodingInfoResolver() {
- _add_map<FieldType::OLAP_FIELD_TYPE_TINYINT, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_TINYINT, FOR_ENCODING, true>();
- _add_map<FieldType::OLAP_FIELD_TYPE_TINYINT, PLAIN_ENCODING>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_SMALLINT, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_SMALLINT, FOR_ENCODING, true>();
- _add_map<FieldType::OLAP_FIELD_TYPE_SMALLINT, PLAIN_ENCODING>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_INT, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_INT, FOR_ENCODING, true>();
- _add_map<FieldType::OLAP_FIELD_TYPE_INT, PLAIN_ENCODING>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_BIGINT, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_BIGINT, FOR_ENCODING, true>();
- _add_map<FieldType::OLAP_FIELD_TYPE_BIGINT, PLAIN_ENCODING>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_UNSIGNED_BIGINT, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_UNSIGNED_INT, BIT_SHUFFLE>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_LARGEINT, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_LARGEINT, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_LARGEINT, FOR_ENCODING, true>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_FLOAT, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_FLOAT, PLAIN_ENCODING>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_DOUBLE, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_DOUBLE, PLAIN_ENCODING>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_CHAR, DICT_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_CHAR, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_CHAR, PLAIN_ENCODING_V2>();
- _add_map<FieldType::OLAP_FIELD_TYPE_CHAR, PREFIX_ENCODING, true>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_VARCHAR, DICT_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_VARCHAR, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_VARCHAR, PLAIN_ENCODING_V2>();
- _add_map<FieldType::OLAP_FIELD_TYPE_VARCHAR, PREFIX_ENCODING, true>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_STRING, DICT_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_STRING, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_STRING, PLAIN_ENCODING_V2>();
- _add_map<FieldType::OLAP_FIELD_TYPE_STRING, PREFIX_ENCODING, true>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_JSONB, DICT_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_JSONB, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_JSONB, PLAIN_ENCODING_V2>();
- _add_map<FieldType::OLAP_FIELD_TYPE_JSONB, PREFIX_ENCODING, true>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_VARIANT, DICT_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_VARIANT, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_VARIANT, PLAIN_ENCODING_V2>();
- _add_map<FieldType::OLAP_FIELD_TYPE_VARIANT, PREFIX_ENCODING, true>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_BOOL, RLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_BOOL, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_BOOL, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_BOOL, PLAIN_ENCODING, true>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_DATE, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_DATE, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_DATE, FOR_ENCODING, true>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_DATEV2, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_DATEV2, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_DATEV2, FOR_ENCODING, true>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_DATETIMEV2, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_DATETIMEV2, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_DATETIMEV2, FOR_ENCODING, true>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_DATETIME, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_DATETIME, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_DATETIME, FOR_ENCODING, true>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_TIMESTAMPTZ, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_TIMESTAMPTZ, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_TIMESTAMPTZ, FOR_ENCODING, true>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_DECIMAL, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_DECIMAL, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_DECIMAL, BIT_SHUFFLE, true>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_DECIMAL32, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_DECIMAL32, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_DECIMAL32, BIT_SHUFFLE, true>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_DECIMAL64, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_DECIMAL64, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_DECIMAL64, BIT_SHUFFLE, true>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_DECIMAL128I, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_DECIMAL128I, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_DECIMAL128I, BIT_SHUFFLE, true>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_DECIMAL256, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_DECIMAL256, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_DECIMAL256, BIT_SHUFFLE, true>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_IPV4, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_IPV4, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_IPV4, BIT_SHUFFLE, true>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_IPV6, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_IPV6, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_IPV6, BIT_SHUFFLE, true>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_HLL, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_HLL, PLAIN_ENCODING_V2>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_BITMAP, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_BITMAP, PLAIN_ENCODING_V2>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_QUANTILE_STATE, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_QUANTILE_STATE, PLAIN_ENCODING_V2>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_AGG_STATE, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_AGG_STATE, PLAIN_ENCODING_V2>();
+ using FT = FieldType;
Review Comment:
这种短名都没啥意义。。。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](BE) split EncodingInfo defaults into 4 explicit maps [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63622:
URL: https://github.com/apache/doris/pull/63622#issuecomment-4533814950
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](BE) split EncodingInfo defaults into 4 explicit maps [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63622:
URL: https://github.com/apache/doris/pull/63622#issuecomment-4534586554
<details>
<summary>TPC-H: <b>Total hot run time: 32287 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b1e1a5698074aa88e6fa9a12ab270e3d63b40a42, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17639 4217 4073 4073
q2 q3 10786 1456 844 844
q4 4692 484 347 347
q5 7594 2375 2200 2200
q6 263 187 144 144
q7 1027 783 655 655
q8 9389 1660 1620 1620
q9 5418 5069 4998 4998
q10 6408 2233 1890 1890
q11 440 273 255 255
q12 648 430 301 301
q13 18111 3408 2794 2794
q14 267 266 251 251
q15 q16 832 780 712 712
q17 984 960 1014 960
q18 7084 5764 5518 5518
q19 1303 1429 1243 1243
q20 598 427 291 291
q21 6229 2866 2868 2866
q22 465 379 325 325
Total cold run time: 100177 ms
Total hot run time: 32287 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4970 4931 4916 4916
q2 q3 5088 5244 4691 4691
q4 2406 2253 1452 1452
q5 4989 4705 4828 4705
q6 239 188 138 138
q7 1903 1772 1581 1581
q8 2524 2175 2173 2173
q9 7506 7563 7471 7471
q10 4776 4757 4290 4290
q11 549 400 371 371
q12 752 749 553 553
q13 3121 3436 2826 2826
q14 270 287 257 257
q15 q16 677 709 609 609
q17 1327 1302 1300 1300
q18 7422 7056 6905 6905
q19 1152 1110 1094 1094
q20 2258 2211 1965 1965
q21 5420 4732 4609 4609
q22 541 484 421 421
Total cold run time: 57890 ms
Total hot run time: 52327 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](BE) split EncodingInfo defaults into 4 explicit maps [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63622:
URL: https://github.com/apache/doris/pull/63622#issuecomment-4534652600
<details>
<summary>TPC-DS: <b>Total hot run time: 173107 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b1e1a5698074aa88e6fa9a12ab270e3d63b40a42, data reload: false
query5 4329 672 523 523
query6 327 221 198 198
query7 4227 595 308 308
query8 327 262 226 226
query9 8850 4146 4125 4125
query10 443 346 303 303
query11 5794 2634 2226 2226
query12 181 128 126 126
query13 1300 675 430 430
query14 6249 5560 5265 5265
query14_1 4566 4547 4505 4505
query15 215 217 188 188
query16 1021 485 397 397
query17 1153 755 626 626
query18 2485 496 369 369
query19 222 218 174 174
query20 145 140 134 134
query21 219 140 124 124
query22 13681 13541 13513 13513
query23 17428 16539 16240 16240
query23_1 16329 16413 16508 16413
query24 7551 1818 1341 1341
query24_1 1321 1350 1361 1350
query25 561 472 426 426
query26 1350 324 182 182
query27 2723 593 342 342
query28 4513 2111 2025 2025
query29 984 628 510 510
query30 311 245 203 203
query31 1131 1085 971 971
query32 91 75 70 70
query33 540 352 296 296
query34 1204 1131 675 675
query35 787 806 697 697
query36 1416 1409 1243 1243
query37 156 113 94 94
query38 3216 3189 3086 3086
query39 955 929 922 922
query39_1 870 872 875 872
query40 226 165 127 127
query41 65 65 63 63
query42 111 110 110 110
query43 337 343 295 295
query44
query45 218 209 199 199
query46 1085 1245 766 766
query47 2416 2409 2278 2278
query48 411 416 317 317
query49 626 496 386 386
query50 972 357 249 249
query51 4361 4327 4414 4327
query52 107 105 96 96
query53 262 292 208 208
query54 319 277 257 257
query55 93 96 85 85
query56 317 298 321 298
query57 1432 1434 1331 1331
query58 300 279 275 275
query59 1626 1668 1398 1398
query60 320 318 315 315
query61 151 151 158 151
query62 692 656 578 578
query63 244 209 207 207
query64 2415 833 668 668
query65
query66 1734 489 364 364
query67 30221 29557 29974 29557
query68
query69 473 337 306 306
query70 1042 1024 1008 1008
query71 305 280 275 275
query72 3030 2772 2441 2441
query73 855 786 435 435
query74 5124 4959 4823 4823
query75 2708 2646 2293 2293
query76 2271 1184 780 780
query77 409 425 350 350
query78 12329 12517 11915 11915
query79 1282 1052 808 808
query80 591 546 454 454
query81 458 279 246 246
query82 238 157 121 121
query83 274 284 252 252
query84 267 144 114 114
query85 844 539 451 451
query86 366 335 349 335
query87 3406 3409 3277 3277
query88 3640 2789 2759 2759
query89 430 397 340 340
query90 2128 188 187 187
query91 179 167 139 139
query92 81 81 84 81
query93 1486 1392 838 838
query94 547 343 306 306
query95 698 472 377 377
query96 1056 834 358 358
query97 2734 2765 2645 2645
query98 241 230 228 228
query99 1201 1144 1023 1023
Total cold run time: 253938 ms
Total hot run time: 173107 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](BE) split EncodingInfo defaults into 4 explicit maps [doris]
Posted by "csun5285 (via GitHub)" <gi...@apache.org>.
csun5285 commented on PR #63622:
URL: https://github.com/apache/doris/pull/63622#issuecomment-4552203360
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](BE) split EncodingInfo defaults into 4 explicit maps [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #63622:
URL: https://github.com/apache/doris/pull/63622#discussion_r3308229250
##########
be/src/storage/segment/encoding_info.cpp:
##########
@@ -214,124 +216,181 @@ struct TypeEncodingTraits<type, PREFIX_ENCODING, Slice> {
};
EncodingInfoResolver::EncodingInfoResolver() {
- _add_map<FieldType::OLAP_FIELD_TYPE_TINYINT, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_TINYINT, FOR_ENCODING, true>();
- _add_map<FieldType::OLAP_FIELD_TYPE_TINYINT, PLAIN_ENCODING>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_SMALLINT, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_SMALLINT, FOR_ENCODING, true>();
- _add_map<FieldType::OLAP_FIELD_TYPE_SMALLINT, PLAIN_ENCODING>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_INT, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_INT, FOR_ENCODING, true>();
- _add_map<FieldType::OLAP_FIELD_TYPE_INT, PLAIN_ENCODING>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_BIGINT, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_BIGINT, FOR_ENCODING, true>();
- _add_map<FieldType::OLAP_FIELD_TYPE_BIGINT, PLAIN_ENCODING>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_UNSIGNED_BIGINT, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_UNSIGNED_INT, BIT_SHUFFLE>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_LARGEINT, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_LARGEINT, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_LARGEINT, FOR_ENCODING, true>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_FLOAT, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_FLOAT, PLAIN_ENCODING>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_DOUBLE, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_DOUBLE, PLAIN_ENCODING>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_CHAR, DICT_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_CHAR, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_CHAR, PLAIN_ENCODING_V2>();
- _add_map<FieldType::OLAP_FIELD_TYPE_CHAR, PREFIX_ENCODING, true>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_VARCHAR, DICT_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_VARCHAR, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_VARCHAR, PLAIN_ENCODING_V2>();
- _add_map<FieldType::OLAP_FIELD_TYPE_VARCHAR, PREFIX_ENCODING, true>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_STRING, DICT_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_STRING, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_STRING, PLAIN_ENCODING_V2>();
- _add_map<FieldType::OLAP_FIELD_TYPE_STRING, PREFIX_ENCODING, true>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_JSONB, DICT_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_JSONB, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_JSONB, PLAIN_ENCODING_V2>();
- _add_map<FieldType::OLAP_FIELD_TYPE_JSONB, PREFIX_ENCODING, true>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_VARIANT, DICT_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_VARIANT, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_VARIANT, PLAIN_ENCODING_V2>();
- _add_map<FieldType::OLAP_FIELD_TYPE_VARIANT, PREFIX_ENCODING, true>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_BOOL, RLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_BOOL, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_BOOL, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_BOOL, PLAIN_ENCODING, true>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_DATE, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_DATE, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_DATE, FOR_ENCODING, true>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_DATEV2, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_DATEV2, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_DATEV2, FOR_ENCODING, true>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_DATETIMEV2, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_DATETIMEV2, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_DATETIMEV2, FOR_ENCODING, true>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_DATETIME, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_DATETIME, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_DATETIME, FOR_ENCODING, true>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_TIMESTAMPTZ, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_TIMESTAMPTZ, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_TIMESTAMPTZ, FOR_ENCODING, true>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_DECIMAL, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_DECIMAL, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_DECIMAL, BIT_SHUFFLE, true>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_DECIMAL32, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_DECIMAL32, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_DECIMAL32, BIT_SHUFFLE, true>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_DECIMAL64, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_DECIMAL64, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_DECIMAL64, BIT_SHUFFLE, true>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_DECIMAL128I, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_DECIMAL128I, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_DECIMAL128I, BIT_SHUFFLE, true>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_DECIMAL256, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_DECIMAL256, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_DECIMAL256, BIT_SHUFFLE, true>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_IPV4, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_IPV4, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_IPV4, BIT_SHUFFLE, true>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_IPV6, BIT_SHUFFLE>();
- _add_map<FieldType::OLAP_FIELD_TYPE_IPV6, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_IPV6, BIT_SHUFFLE, true>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_HLL, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_HLL, PLAIN_ENCODING_V2>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_BITMAP, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_BITMAP, PLAIN_ENCODING_V2>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_QUANTILE_STATE, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_QUANTILE_STATE, PLAIN_ENCODING_V2>();
-
- _add_map<FieldType::OLAP_FIELD_TYPE_AGG_STATE, PLAIN_ENCODING>();
- _add_map<FieldType::OLAP_FIELD_TYPE_AGG_STATE, PLAIN_ENCODING_V2>();
+ // ===== Phase 1: register every supported (type, encoding) combination exactly once =====
+ // _register_supported_encoding CHECKs against duplicates; the Phase 2 calls below do not insert into _encoding_map,
+ // so every (type, encoding) used as a default must appear here first.
+
+ // signed integers
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_TINYINT, BIT_SHUFFLE>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_TINYINT, PLAIN_ENCODING>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_TINYINT, FOR_ENCODING>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_SMALLINT, BIT_SHUFFLE>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_SMALLINT, PLAIN_ENCODING>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_SMALLINT, FOR_ENCODING>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_INT, BIT_SHUFFLE>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_INT, PLAIN_ENCODING>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_INT, FOR_ENCODING>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_BIGINT, BIT_SHUFFLE>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_BIGINT, PLAIN_ENCODING>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_BIGINT, FOR_ENCODING>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_LARGEINT, BIT_SHUFFLE>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_LARGEINT, PLAIN_ENCODING>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_LARGEINT, FOR_ENCODING>();
+
+ // unsigned integers
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_UNSIGNED_BIGINT, BIT_SHUFFLE>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_UNSIGNED_INT, BIT_SHUFFLE>();
+
+ // FLOAT / DOUBLE
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_FLOAT, BIT_SHUFFLE>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_FLOAT, PLAIN_ENCODING>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_DOUBLE, BIT_SHUFFLE>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_DOUBLE, PLAIN_ENCODING>();
+
+ // binary types (CHAR/VARCHAR/STRING/JSONB/VARIANT)
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_CHAR, DICT_ENCODING>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_CHAR, PLAIN_ENCODING>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_CHAR, PLAIN_ENCODING_V2>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_CHAR, PREFIX_ENCODING>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_VARCHAR, DICT_ENCODING>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_VARCHAR, PLAIN_ENCODING>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_VARCHAR, PLAIN_ENCODING_V2>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_VARCHAR, PREFIX_ENCODING>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_STRING, DICT_ENCODING>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_STRING, PLAIN_ENCODING>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_STRING, PLAIN_ENCODING_V2>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_STRING, PREFIX_ENCODING>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_JSONB, DICT_ENCODING>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_JSONB, PLAIN_ENCODING>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_JSONB, PLAIN_ENCODING_V2>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_JSONB, PREFIX_ENCODING>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_VARIANT, DICT_ENCODING>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_VARIANT, PLAIN_ENCODING>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_VARIANT, PLAIN_ENCODING_V2>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_VARIANT, PREFIX_ENCODING>();
+
+ // BOOL
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_BOOL, RLE>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_BOOL, BIT_SHUFFLE>();
+ _register_supported_encoding<FieldType::OLAP_FIELD_TYPE_BOOL, PLAIN_ENCODING>();
+
Review Comment:
这个漏了 _add_map<FieldType::OLAP_FIELD_TYPE_BOOL, PLAIN_ENCODING, true>();
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](BE) split EncodingInfo defaults into 4 explicit maps [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63622:
URL: https://github.com/apache/doris/pull/63622#issuecomment-4561732214
# BE UT Coverage Report
Increment line coverage `91.75% (278/303)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/8eff36afaba543f4d0dca6550f163b964dd15be9_8eff36afaba543f4d0dca6550f163b964dd15be9/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/8eff36afaba543f4d0dca6550f163b964dd15be9_8eff36afaba543f4d0dca6550f163b964dd15be9/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 53.99% (20979/38856) |
| Line Coverage | 37.53% (198799/529771) |
| Region Coverage | 33.80% (155767/460885) |
| Branch Coverage | 34.79% (67790/194846) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](BE) split EncodingInfo defaults into 4 explicit maps [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63622:
URL: https://github.com/apache/doris/pull/63622#issuecomment-4562407105
<details>
<summary>TPC-H: <b>Total hot run time: 31806 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8eff36afaba543f4d0dca6550f163b964dd15be9, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 16666 4094 4069 4069
q2 q3 10440 1413 833 833
q4 4668 466 344 344
q5 8070 2277 2120 2120
q6 244 177 139 139
q7 978 781 669 669
q8 9361 1724 1577 1577
q9 5116 4985 4953 4953
q10 6381 2227 1884 1884
q11 439 270 244 244
q12 645 425 296 296
q13 18109 3310 2816 2816
q14 273 256 246 246
q15 q16 797 771 715 715
q17 961 915 857 857
q18 6968 5853 5545 5545
q19 1179 1307 1172 1172
q20 509 400 264 264
q21 6375 2829 2733 2733
q22 458 375 330 330
Total cold run time: 98637 ms
Total hot run time: 31806 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 5048 4849 4727 4727
q2 q3 4980 5248 4743 4743
q4 2136 2250 1393 1393
q5 4792 4837 4663 4663
q6 231 177 125 125
q7 1861 1829 1560 1560
q8 2406 2143 2209 2143
q9 8008 7769 7315 7315
q10 4749 4710 4286 4286
q11 535 379 354 354
q12 732 749 526 526
q13 2979 3368 2777 2777
q14 277 280 260 260
q15 q16 702 700 619 619
q17 1296 1266 1262 1262
q18 7341 6792 6881 6792
q19 1117 1124 1121 1121
q20 2237 2233 1950 1950
q21 5343 4607 4497 4497
q22 526 468 405 405
Total cold run time: 57296 ms
Total hot run time: 51518 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](BE) split EncodingInfo defaults into 4 explicit maps [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63622:
URL: https://github.com/apache/doris/pull/63622#issuecomment-4562519112
# BE Regression && UT Coverage Report
Increment line coverage `91.32% (284/311)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63622_8eff36afaba543f4d0dca6550f163b964dd15be9_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63622_8eff36afaba543f4d0dca6550f163b964dd15be9_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 72.06% (27424/38055) |
| Line Coverage | 55.39% (292711/528424) |
| Region Coverage | 52.30% (243372/465306) |
| Branch Coverage | 53.64% (104900/195576) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](BE) split EncodingInfo defaults into 4 explicit maps [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63622:
URL: https://github.com/apache/doris/pull/63622#issuecomment-4561539058
# Cloud UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/8eff36afaba543f4d0dca6550f163b964dd15be9_8eff36afaba543f4d0dca6550f163b964dd15be9_cloud/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/8eff36afaba543f4d0dca6550f163b964dd15be9_8eff36afaba543f4d0dca6550f163b964dd15be9_cloud/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 78.06% (1854/2375) |
| Line Coverage | 64.52% (33332/51665) |
| Region Coverage | 65.19% (16521/25343) |
| Branch Coverage | 55.75% (8838/15854) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](BE) split EncodingInfo defaults into 4 explicit maps [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on PR #63622:
URL: https://github.com/apache/doris/pull/63622#issuecomment-4569881818
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org