You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "morningman (via GitHub)" <gi...@apache.org> on 2026/10/23 12:57:41 UTC
[PR] [fix](be) Keep projected structs readable with nested Iceberg equality keys in V1 Parquet [doris]
morningman opened a new pull request, #69547:
URL: https://github.com/apache/doris/pull/69547
### What problem does this PR solve?
Issue Number: None
Related PR: #66538
Problem Summary:
**In short.** This ports a fix from branch-4.1 #66538 (Iceberg V3 default values) to the V1 Parquet reader on master. When an Iceberg equality delete is keyed on a struct field such as `payload.existing`, and the query also projects `payload` (any `SELECT *`), or the delete has several keys in the same struct, master reads only one of the columns that map to the struct, and returns an empty struct or no rows.
**Background.** The V1 Parquet reader reads a nested equality key through its top-level struct. A hidden carrier column named `__equality_delete_column__<id>_<leaf>` maps to the physical root (`payload`), the whole struct is read into it, and the key's leaf is extracted after the read. `ParquetReader::_init_read_columns` builds a map from each file column to one table column.
**The problem, and what it cost.** When two table columns name the same file column, the later one replaces the earlier one in that map, and only one of them gets a column reader. Two cases do this: a projected `payload` next to the carrier of `payload.existing`, and the carriers of two keys under one root, such as `payload.a` and `payload.b`. Reproduced with BE unit tests on a file holding (1,{10}), (2,{20}), (3,{30}) and an equality delete on `payload.existing = 20`:
| Query projects | Master today | After this PR |
|---|---|---|
| `id, payload` | `payload` is empty, and the block has fewer `payload` rows than `id` rows (a DCHECK in `vparquet_group_reader.cpp` under ASAN) | (1,{10}), (3,{30}) |
| `payload, id` | no rows | (1,{10}), (3,{30}) |
| `id`, with keys `payload.existing` and `payload.k` | the carrier of `payload.existing` is never read | rows 1 and 3 |
ORC reads of a projected root struct are not affected: the ORC cases below pass without this change.
**How this PR fixes it.** It follows branch-4.1 / branch-4.2 (the Parquet carrier setup in `be/src/format/table/iceberg_reader.cpp` and `ParquetReader::set_duplicate_file_column_aliases`):
- When the query projects the key's root struct, the file has the key's full path and the projected struct holds the key with the key's type, the key is taken from the projected column. Nothing extra is read.
- Otherwise the first key under a physical root reads the root, and every later key under that root is extracted from that carrier's column.
- A carrier that reads nothing gets a placeholder of the batch row count, so the reader filters it with the other columns, and the leaf replaces it after the read. All leaves are extracted before any carrier column is replaced, because one carrier can be the source of other keys.
- `ParquetReader` reads a file column once for each table column that names it, and prefetches it once. This covers a key that the projected struct lacks, for example a key dropped from the table, whose carrier reads the same root as the projected `payload`.
Two differences from branch-4.x, both on the safe side: a key whose path is incomplete in the file (an older file written before the key existed) keeps reading the physical root instead of relying on the projected struct for the missing child, and a key whose type differs from the projected type (a key written before a type promotion) also keeps reading the physical root. Branch-4.x casts that case; master does not cast equality keys yet, which is a separate fix.
**Results.** All 50 tests of `IcebergReaderTest` pass. Without this change, each of the five Parquet cases below aborts on the DCHECK `block->rows() == col.column->size()` in `vparquet_group_reader.cpp`, because one of the columns that share the root is never read:
- Four cases from #66538 for a projected root struct, Parquet and ORC, with `payload` listed before and after `id`. Without this change the Parquet cases leave `payload` unread in both orders; the ORC cases already pass.
- `v1_parquet_equality_keys_share_unprojected_root`: two keys under an unprojected root share one carrier. Without this change the carrier of `payload.existing` is never read.
- `v1_parquet_equality_keys_come_from_projected_root`: two keys are taken from the projected root. Without this change `payload` is never read.
- `v1_parquet_equality_key_outside_projected_struct_rereads_root`: the projected struct lacks the key, so the carrier reads the same file column again. Without this change `payload` is never read.
Original author: @hubgeter
### Release note
Fix empty or missing rows when the V1 Parquet reader reads an Iceberg table with an equality delete keyed on a struct field, and the query also projects that struct or the delete has several keys in the struct.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [x] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [x] Yes. <!-- Explain the behavior change -->
Such reads return the correct rows.
- Does this need documentation?
- [x] No.
- [ ] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Keep projected structs readable with nested Iceberg equality keys in V1 Parquet [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #69547:
URL: https://github.com/apache/doris/pull/69547#issuecomment-6383400186
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Keep projected structs readable with nested Iceberg equality keys in V1 Parquet [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #69547:
URL: https://github.com/apache/doris/pull/69547#issuecomment-6383400539
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Keep projected structs readable with nested Iceberg equality keys in V1 Parquet [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #69547:
URL: https://github.com/apache/doris/pull/69547#issuecomment-6365417826
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Keep projected structs readable with nested Iceberg equality keys in V1 Parquet [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #69547:
URL: https://github.com/apache/doris/pull/69547#issuecomment-6365416597
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Keep projected structs readable with nested Iceberg equality keys in V1 Parquet [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman merged PR #69547:
URL: https://github.com/apache/doris/pull/69547
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Keep projected structs readable with nested Iceberg equality keys in V1 Parquet [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #69547:
URL: https://github.com/apache/doris/pull/69547#discussion_r4365360556
##########
be/src/format/table/iceberg_reader.cpp:
##########
@@ -570,6 +668,44 @@ Status IcebergParquetReader::on_before_init_reader(ReaderInitContext* ctx) {
}
return nullptr;
};
+ std::unordered_map<int32_t, const SlotDescriptor*> projected_slots;
+ for (const auto* slot : ctx->tuple_descriptor->slots()) {
+ projected_slots.emplace(slot->col_unique_id(), slot);
+ }
+ // Finds a nested key inside its root struct when the query projects that root, so the key is
+ // taken from the projected column instead of reading the root a second time.
+ const auto find_projected_root = [&](int32_t field_id, const DataTypePtr& key_type,
+ std::string* root_name,
+ std::vector<size_t>* child_indexes) -> Status {
+ const auto key_path = _find_schema_field_path(field_id);
+ if (key_path.empty()) {
+ return Status::OK();
+ }
+ DORIS_CHECK(key_path.front()->__isset.id);
+ const auto slot = projected_slots.find(key_path.front()->id);
+ if (slot == projected_slots.end()) {
+ return Status::OK();
+ }
+ std::vector<const schema::external::TField*> root_path;
+ if (!_find_schema_field_path_in_root(_current_schema_root(), key_path.front()->id,
+ &root_path)) {
+ return Status::OK();
+ }
+ DORIS_CHECK(root_path.size() == 1);
+ DataTypePtr leaf_type;
+ bool found = false;
+ RETURN_IF_ERROR(find_projected_equality_key_path(*root_path.front(), key_path,
+ slot->second->type(), child_indexes,
+ &leaf_type, &found));
+ // The projected leaf has the table's current type. A key written before a type promotion
+ // (INT promoted to BIGINT) keeps its own type, so it still reads the physical root.
+ if (found && remove_nullable(leaf_type)->equals(*remove_nullable(key_type))) {
Review Comment:
[P1] Verify physical binding before reusing a projected key
For a legacy FE plan without `iceberg_scan_semantics_version`, the projected struct and the hidden key can resolve different physical fields. For example, let the file contain ID-tagged roots `id#1` and `payload#2`, with children `old_key#3 INT` and an unrelated ID-less sibling; the current projected type is `payload<new_key#3 INT>`, with no name-mapping aliases. Hidden-key lookup finds field 3 by ID, but `BuildTableInfoUtil::by_parquet_field_id_with_name_mapping` switches the projected struct's children to name matching when that sibling lacks an ID (`table_schema_change_helper.cpp:895-921`), so `new_key` is filled with NULL. This logical-path/type check still selects `payload`, and the reuse branch skips the physical carrier. An equality delete on field 3 = 20 then compares NULL and silently retains the deleted row. Older FE source actually emits nested schema metadata without the marker, so this is reachable during a BE-first rolling upgrade. Please verify that the projected sc
hema-mapping path resolves the same physical leaf as `file_path`, or fall back to the carrier, and cover this legacy mixed-ID/renamed-key case. The new tests all set semantics v2 and miss this branch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Keep projected structs readable with nested Iceberg equality keys in V1 Parquet [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #69547:
URL: https://github.com/apache/doris/pull/69547#issuecomment-6383399865
Addressed the second review round in `9858972766f`:
- 4365360556 (P1, a legacy plan binds the projected struct by name and the hidden key by field ID): fixed. A nested key is taken from the projected root only when the projected column's mapping reads the same file field at every level of the key's physical path; otherwise the key reads its own carrier. New test `v1_parquet_legacy_plan_key_rereads_root_when_projected_child_matches_by_name`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Keep projected structs readable with nested Iceberg equality keys in V1 Parquet [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69547:
URL: https://github.com/apache/doris/pull/69547#issuecomment-6365411662
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://doris.apache.org/community/how-to-contribute/review-and-ci).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Keep projected structs readable with nested Iceberg equality keys in V1 Parquet [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69547:
URL: https://github.com/apache/doris/pull/69547#issuecomment-6385333175
# BE Regression && UT Coverage Report
Increment line coverage `91.93% (205/223)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69547_9858972766f2fcb75452e2e887507ae9e2c631b9_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69547_9858972766f2fcb75452e2e887507ae9e2c631b9_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 77.29% (35983/46558) |
| Line Coverage | 62.60% (406858/649977) |
| Region Coverage | 59.12% (344233/582298) |
| Branch Coverage | 60.02% (159030/264973) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Keep projected structs readable with nested Iceberg equality keys in V1 Parquet [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #69547:
URL: https://github.com/apache/doris/pull/69547#discussion_r4359868360
##########
be/src/format/table/iceberg_reader_mixin.h:
##########
@@ -1226,6 +1245,22 @@ Status IcebergReaderMixin<BaseReader>::_register_missing_equality_delete_column(
return Status::OK();
}
+template <typename BaseReader>
+void IcebergReaderMixin<BaseReader>::_register_derived_equality_delete_column(
+ const std::string& name, const DataTypePtr& delete_key_type) {
+ DORIS_CHECK(delete_key_type != nullptr);
+ // The key is extracted from its source column after the read. Until then a placeholder keeps
+ // the carrier at the batch row count, so the reader filters it with the other columns.
+ auto placeholder = delete_key_type->create_column();
+ placeholder->insert_default();
+ this->register_synthesized_column_handler(
Review Comment:
[P1] Resolve hidden placeholders without requiring tuple slots
The new projected-root and shared-carrier paths register internal names such as `__equality_delete_column__2_existing` here, but those keys only exist in `_expand_columns`, not the query tuple. `ParquetReader::_do_init_reader` unconditionally calls `_classify_columns_for_lazy_read`, whose production-only synthesized-column loop requires a matching tuple slot. This aborts on `DCHECK(slot_id != -1)` in debug builds. In release, `RowDescriptor::get_column_id(-1)` returns `-1`, which is stored as `UINT32_MAX` in `all_predicate_col_ids`; with lazy materialization and a partially selective predicate (for example `SELECT id, payload ... WHERE id > 1` with a nested equality delete), `_do_lazy_read` passes that index to `Block::filter_block_internal` and accesses outside the block. The added tests miss this because `BE_TEST` excludes the classifier loop. Please resolve these internal keys through their actual expanded block positions, or give them a separate fill/filter path, and cover pro
duction classification with a selective predicate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Keep projected structs readable with nested Iceberg equality keys in V1 Parquet [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69547:
URL: https://github.com/apache/doris/pull/69547#issuecomment-6379484090
# BE Regression && UT Coverage Report
Increment line coverage `92.00% (184/200)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69547_5acba3e2c976c12a28fb9098cbcd08b1b697669d_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69547_5acba3e2c976c12a28fb9098cbcd08b1b697669d_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 77.30% (35987/46557) |
| Line Coverage | 62.61% (406922/649952) |
| Region Coverage | 59.12% (344217/582279) |
| Branch Coverage | 60.04% (159092/264955) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Keep projected structs readable with nested Iceberg equality keys in V1 Parquet [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69547:
URL: https://github.com/apache/doris/pull/69547#issuecomment-6383779352
<details>
<summary>TPC-H: <b>Total hot run time: 28247 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9858972766f2fcb75452e2e887507ae9e2c631b9, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17590 5165 5048 5048
q2 2108 316 280 280
q3 10225 1379 821 821
q4 4679 454 334 334
q5 7430 785 523 523
q6 184 183 151 151
q7 741 775 594 594
q8 9253 1343 1053 1053
q9 5538 4185 4173 4173
q10 6820 1283 1024 1024
q11 436 243 222 222
q12 639 387 275 275
q13 18091 3682 3011 3011
q14 272 273 242 242
q15 q16 734 717 666 666
q17 1194 764 571 571
q18 7143 6160 6079 6079
q19 1265 950 594 594
q20 395 346 226 226
q21 5478 2116 2302 2116
q22 346 297 244 244
Total cold run time: 100561 ms
Total hot run time: 28247 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 5634 5585 5585 5585
q2 667 569 518 518
q3 5207 5322 4962 4962
q4 1980 2133 1528 1528
q5 5286 5135 5587 5135
q6 263 202 154 154
q7 2310 2114 1898 1898
q8 3105 2866 2654 2654
q9 7952 7923 7885 7885
q10 3777 3716 3247 3247
q11 605 434 439 434
q12 711 702 513 513
q13 3544 3789 3382 3382
q14 300 307 281 281
q15 q16 707 708 644 644
q17 8147 7505 7394 7394
q18 12962 12208 12871 12208
q19 929 774 784 774
q20 2137 2129 1934 1934
q21 5575 4581 4692 4581
q22 505 455 423 423
Total cold run time: 72303 ms
Total hot run time: 66134 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Keep projected structs readable with nested Iceberg equality keys in V1 Parquet [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69547:
URL: https://github.com/apache/doris/pull/69547#issuecomment-6377984796
<details>
<summary>TPC-DS: <b>Total hot run time: 151020 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5acba3e2c976c12a28fb9098cbcd08b1b697669d, data reload: false
query5 4306 577 441 441
query6 426 209 190 190
query7 4812 441 221 221
query8 320 169 149 149
query9 8767 3150 3163 3150
query10 485 346 270 270
query11 5923 3898 3577 3577
query12 145 92 86 86
query13 1262 460 335 335
query14 6164 3967 3663 3663
query14_1 3457 3454 3411 3411
query15 212 203 185 185
query16 937 441 409 409
query17 861 672 538 538
query18 2424 444 310 310
query19 203 172 143 143
query20 87 84 87 84
query21 212 131 109 109
query22 12977 13103 12794 12794
query23 13510 12903 12506 12506
query23_1 12639 12554 12499 12499
query24 7390 821 464 464
query24_1 482 479 476 476
query25 529 429 367 367
query26 1262 254 146 146
query27 2773 443 274 274
query28 4635 1964 1923 1923
query29 1570 624 459 459
query30 296 225 184 184
query31 1007 960 820 820
query32 143 98 93 93
query33 536 313 259 259
query34 957 843 503 503
query35 745 759 639 639
query36 814 836 722 722
query37 124 99 87 87
query38 1779 1749 1682 1682
query39 798 796 757 757
query39_1 719 737 749 737
query40 218 117 99 99
query41 72 71 68 68
query42 88 89 83 83
query43 352 362 315 315
query44 1346 693 696 693
query45 194 178 168 168
query46 823 977 624 624
query47 2937 3014 2807 2807
query48 293 319 213 213
query49 574 426 294 294
query50 658 285 199 199
query51 10486 10289 10257 10257
query52 81 82 72 72
query53 185 212 151 151
query54 243 193 176 176
query55 79 69 65 65
query56 240 209 199 199
query57 1675 1621 1561 1561
query58 280 261 260 260
query59 2159 2098 1959 1959
query60 293 249 264 249
query61 145 146 154 146
query62 389 332 279 279
query63 189 166 163 163
query64 2736 942 755 755
query65 3357 3290 3302 3290
query66 1783 421 301 301
query67 20075 20172 19925 19925
query68 2993 1006 618 618
query69 386 288 253 253
query70 841 797 769 769
query71 297 235 212 212
query72 2748 2583 2283 2283
query73 513 520 288 288
query74 4655 4543 4378 4378
query75 2345 2299 1948 1948
query76 2319 1028 601 601
query77 350 370 261 261
query78 9716 9763 9244 9244
query79 964 913 524 524
query80 662 425 362 362
query81 539 316 269 269
query82 577 130 107 107
query83 258 198 181 181
query84 315 121 103 103
query85 841 424 381 381
query86 324 233 230 230
query87 1982 1976 1810 1810
query88 3748 2686 2646 2646
query89 331 281 246 246
query90 1828 162 176 162
query91 167 153 130 130
query92 96 89 88 88
query93 1008 1018 576 576
query94 464 283 308 283
query95 608 332 289 289
query96 671 484 233 233
query97 2485 2449 2318 2318
query98 159 151 150 150
query99 772 769 646 646
Total cold run time: 234788 ms
Total hot run time: 151020 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Keep projected structs readable with nested Iceberg equality keys in V1 Parquet [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on code in PR #69547:
URL: https://github.com/apache/doris/pull/69547#discussion_r4365298253
##########
be/src/format/table/iceberg_reader_mixin.h:
##########
@@ -1226,6 +1245,22 @@ Status IcebergReaderMixin<BaseReader>::_register_missing_equality_delete_column(
return Status::OK();
}
+template <typename BaseReader>
+void IcebergReaderMixin<BaseReader>::_register_derived_equality_delete_column(
+ const std::string& name, const DataTypePtr& delete_key_type) {
+ DORIS_CHECK(delete_key_type != nullptr);
+ // The key is extracted from its source column after the read. Until then a placeholder keeps
+ // the carrier at the batch row count, so the reader filters it with the other columns.
+ auto placeholder = delete_key_type->create_column();
+ placeholder->insert_default();
+ this->register_synthesized_column_handler(
Review Comment:
Fixed in 5acba3e2c97. The keys that the reader does not read — a key extracted from the projected root struct, and the second key under a physical root that another key's carrier reads — are no longer synthesized columns. The reader block has no column for them during the read, so the lazy-read classifier never sees a name without a tuple slot, and `on_after_read_block` appends them once the batch has its final row count (`_materialize_nested_equality_delete_columns` appends the extracted leaf).
The missing-key carrier from #65502 had the same problem, and not only on Parquet: its synthesized handler appended the column after the reader had taken its column count, and the lazy reads of both Parquet and ORC erase columns beyond that count after filtering, so the key's block position went stale. It is now appended after the read as well, on both formats.
Tests: lazy-read variants with a filter on `id` for a key taken from the projected root (both column orders), for keys sharing an unprojected root and for keys taken from the projected root, plus a missing key with and without the filter on Parquet and ORC. Without the fix the four nested cases abort at `vparquet_group_reader.cpp:829` (`column_size == cz`, 2 vs. 3) and both lazy missing-key cases fail with `invalid input position, position=2`; the production slot lookup itself is compiled out under BE_TEST. On reachability: master FE sets `parquet_timestamp_semantics_version = 1` and Iceberg scan semantics v2, which routes Iceberg scans to FileScannerV2, so the V1 readers are reached when a BE that was upgraded first serves a plan of an older FE.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Keep projected structs readable with nested Iceberg equality keys in V1 Parquet [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69547:
URL: https://github.com/apache/doris/pull/69547#issuecomment-6368381399
<details>
<summary>TPC-H: <b>Total hot run time: 28660 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6e2e3711b375b242ad91f8bce250e8d94e93a3e5, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17575 5178 5024 5024
q2 2123 325 274 274
q3 10204 1401 812 812
q4 4682 460 340 340
q5 7424 806 531 531
q6 187 181 150 150
q7 745 794 618 618
q8 9260 1363 1057 1057
q9 5567 4201 4164 4164
q10 6811 1298 1028 1028
q11 453 248 225 225
q12 633 391 277 277
q13 18152 3624 3017 3017
q14 277 277 250 250
q15 q16 726 719 671 671
q17 1204 771 576 576
q18 7080 6183 6138 6138
q19 1109 954 603 603
q20 397 335 220 220
q21 5401 2420 2427 2420
q22 390 337 265 265
Total cold run time: 100400 ms
Total hot run time: 28660 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 6005 5886 6051 5886
q2 718 617 594 594
q3 5450 5685 5201 5201
q4 2043 2204 1573 1573
q5 5833 5737 5743 5737
q6 252 195 148 148
q7 2260 2008 1846 1846
q8 3050 2643 2642 2642
q9 8044 7915 7881 7881
q10 3657 3566 3129 3129
q11 586 399 377 377
q12 671 663 505 505
q13 3227 3622 3061 3061
q14 283 287 260 260
q15 q16 663 682 615 615
q17 7764 7002 7056 7002
q18 12971 12240 12864 12240
q19 894 782 776 776
q20 2146 2132 1953 1953
q21 5564 4632 4673 4632
q22 504 480 396 396
Total cold run time: 72585 ms
Total hot run time: 66454 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Keep projected structs readable with nested Iceberg equality keys in V1 Parquet [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69547:
URL: https://github.com/apache/doris/pull/69547#issuecomment-6368577740
<details>
<summary>TPC-DS: <b>Total hot run time: 150622 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6e2e3711b375b242ad91f8bce250e8d94e93a3e5, data reload: false
query5 4315 580 433 433
query6 431 208 187 187
query7 4801 446 223 223
query8 315 170 152 152
query9 8787 3050 3042 3042
query10 472 310 261 261
query11 6073 3865 3570 3570
query12 166 91 86 86
query13 1267 422 305 305
query14 6163 3991 3717 3717
query14_1 3429 3455 3484 3455
query15 204 195 192 192
query16 956 439 426 426
query17 874 662 545 545
query18 2419 409 301 301
query19 187 154 133 133
query20 83 81 83 81
query21 215 125 106 106
query22 12998 13000 12776 12776
query23 13350 12902 12294 12294
query23_1 12628 12420 12561 12420
query24 7374 810 456 456
query24_1 457 457 452 452
query25 508 394 344 344
query26 1247 260 134 134
query27 2778 439 264 264
query28 4636 1922 1913 1913
query29 1554 599 424 424
query30 309 220 189 189
query31 1010 920 818 818
query32 145 106 101 101
query33 536 298 225 225
query34 897 836 500 500
query35 733 745 640 640
query36 807 813 724 724
query37 118 98 76 76
query38 1788 1728 1675 1675
query39 789 789 742 742
query39_1 727 724 735 724
query40 235 109 96 96
query41 66 65 66 65
query42 83 83 81 81
query43 339 366 313 313
query44 1286 668 687 668
query45 182 184 179 179
query46 822 1002 557 557
query47 2969 3075 2659 2659
query48 294 313 223 223
query49 579 401 295 295
query50 667 274 210 210
query51 10598 10537 10302 10302
query52 85 81 71 71
query53 188 209 156 156
query54 260 220 186 186
query55 79 69 65 65
query56 242 241 232 232
query57 1667 1627 1571 1571
query58 288 292 263 263
query59 2121 2179 1940 1940
query60 297 247 240 240
query61 170 157 169 157
query62 400 335 283 283
query63 193 158 162 158
query64 2857 1041 879 879
query65 3357 3306 3278 3278
query66 1815 430 299 299
query67 20190 20377 20030 20030
query68 3214 983 605 605
query69 395 286 248 248
query70 861 781 801 781
query71 310 231 208 208
query72 2864 2616 2346 2346
query73 518 519 295 295
query74 4678 4500 4369 4369
query75 2352 2319 1915 1915
query76 2386 1007 581 581
query77 363 351 275 275
query78 9715 9819 9182 9182
query79 992 825 528 528
query80 1092 418 360 360
query81 608 318 272 272
query82 567 133 107 107
query83 305 199 189 189
query84 307 124 103 103
query85 874 422 382 382
query86 375 239 232 232
query87 1993 1958 1820 1820
query88 3608 2682 2650 2650
query89 331 277 245 245
query90 1861 172 175 172
query91 164 150 133 133
query92 96 91 94 91
query93 984 1008 577 577
query94 589 291 258 258
query95 576 374 303 303
query96 656 492 227 227
query97 2463 2427 2333 2333
query98 159 149 147 147
query99 762 757 637 637
Total cold run time: 235645 ms
Total hot run time: 150622 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Keep projected structs readable with nested Iceberg equality keys in V1 Parquet [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #69547:
URL: https://github.com/apache/doris/pull/69547#issuecomment-6377509354
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Keep projected structs readable with nested Iceberg equality keys in V1 Parquet [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #69547:
URL: https://github.com/apache/doris/pull/69547#issuecomment-6377508944
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Keep projected structs readable with nested Iceberg equality keys in V1 Parquet [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69547:
URL: https://github.com/apache/doris/pull/69547#issuecomment-6377879472
<details>
<summary>TPC-H: <b>Total hot run time: 28624 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5acba3e2c976c12a28fb9098cbcd08b1b697669d, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17585 5149 5085 5085
q2 2164 314 260 260
q3 10214 1376 841 841
q4 4680 465 343 343
q5 7444 830 545 545
q6 212 184 154 154
q7 756 797 608 608
q8 9263 1383 1088 1088
q9 5497 4221 4192 4192
q10 6847 1284 1023 1023
q11 442 251 228 228
q12 636 420 279 279
q13 18068 3583 3030 3030
q14 270 275 244 244
q15 q16 723 721 650 650
q17 1193 744 578 578
q18 6888 6173 6107 6107
q19 1099 967 618 618
q20 389 349 225 225
q21 4871 2260 2338 2260
q22 354 321 266 266
Total cold run time: 99595 ms
Total hot run time: 28624 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 5616 5612 5621 5612
q2 673 609 545 545
q3 5203 5410 4910 4910
q4 2000 2143 1564 1564
q5 5245 5163 5570 5163
q6 253 190 168 168
q7 2320 2083 1840 1840
q8 3063 2617 2679 2617
q9 8107 7975 7800 7800
q10 3780 3702 3227 3227
q11 612 580 404 404
q12 701 691 528 528
q13 3525 3876 3228 3228
q14 334 330 296 296
q15 q16 752 743 679 679
q17 8170 7708 6975 6975
q18 13077 12222 12892 12222
q19 923 839 835 835
q20 2153 2129 1918 1918
q21 5709 4686 4747 4686
q22 491 454 410 410
Total cold run time: 72707 ms
Total hot run time: 65627 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Keep projected structs readable with nested Iceberg equality keys in V1 Parquet [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on code in PR #69547:
URL: https://github.com/apache/doris/pull/69547#discussion_r4366910227
##########
be/src/format/table/iceberg_reader.cpp:
##########
@@ -570,6 +668,44 @@ Status IcebergParquetReader::on_before_init_reader(ReaderInitContext* ctx) {
}
return nullptr;
};
+ std::unordered_map<int32_t, const SlotDescriptor*> projected_slots;
+ for (const auto* slot : ctx->tuple_descriptor->slots()) {
+ projected_slots.emplace(slot->col_unique_id(), slot);
+ }
+ // Finds a nested key inside its root struct when the query projects that root, so the key is
+ // taken from the projected column instead of reading the root a second time.
+ const auto find_projected_root = [&](int32_t field_id, const DataTypePtr& key_type,
+ std::string* root_name,
+ std::vector<size_t>* child_indexes) -> Status {
+ const auto key_path = _find_schema_field_path(field_id);
+ if (key_path.empty()) {
+ return Status::OK();
+ }
+ DORIS_CHECK(key_path.front()->__isset.id);
+ const auto slot = projected_slots.find(key_path.front()->id);
+ if (slot == projected_slots.end()) {
+ return Status::OK();
+ }
+ std::vector<const schema::external::TField*> root_path;
+ if (!_find_schema_field_path_in_root(_current_schema_root(), key_path.front()->id,
+ &root_path)) {
+ return Status::OK();
+ }
+ DORIS_CHECK(root_path.size() == 1);
+ DataTypePtr leaf_type;
+ bool found = false;
+ RETURN_IF_ERROR(find_projected_equality_key_path(*root_path.front(), key_path,
+ slot->second->type(), child_indexes,
+ &leaf_type, &found));
+ // The projected leaf has the table's current type. A key written before a type promotion
+ // (INT promoted to BIGINT) keeps its own type, so it still reads the physical root.
+ if (found && remove_nullable(leaf_type)->equals(*remove_nullable(key_type))) {
Review Comment:
Confirmed, and fixed in `9858972766f`. With a plan from an FE that predates `iceberg_scan_semantics_version`, the projected struct is mapped level by level, and a level whose file children do not all carry field IDs is matched by name (kept for old FE plans by #65784), while the hidden key is located by field ID. `find_projected_root` checked only the key's logical path and type, so for a file `payload{existing without an ID, k#3}` with `k` renamed it took the key from the projected `renamed_k`, which that mapping reads as NULL. Before taking a key from the projected root, the reader now checks that the projected column's schema mapping resolves the root and every level of the key's path to the same file fields as the key's physical path, the way the struct reader resolves each child (`children_file_column_name`); when it does not, the key reads its own carrier, which the multimap change already supports. `v1_parquet_legacy_plan_key_rereads_root_when_projected_child_matches_by_nam
e` covers this file under a legacy plan (without the check, id 2 survives the delete) and under a plan with the marker, where both sides bind `k#3` by field ID and the key still comes from the projected payload.
On reachability: plans that carry the marker resolve both sides by field ID, and current FEs also send the Parquet timestamp marker, which routes Iceberg scans to FileScannerV2. So this case needs an FE at 4.1.3 or older during a BE-first upgrade, plus a data file whose nested struct mixes ID-tagged and ID-less children, which Iceberg writers, Doris's included, do not produce. branch-4.1 (since 4.1.4) and branch-4.2 resolve the projected root the same way; we will report it there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Keep projected structs readable with nested Iceberg equality keys in V1 Parquet [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69547:
URL: https://github.com/apache/doris/pull/69547#issuecomment-6372583336
# BE Regression && UT Coverage Report
Increment line coverage `91.84% (180/196)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69547_6e2e3711b375b242ad91f8bce250e8d94e93a3e5_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69547_6e2e3711b375b242ad91f8bce250e8d94e93a3e5_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 68.59% (31935/46557) |
| Line Coverage | 53.28% (346283/649954) |
| Region Coverage | 48.90% (284732/582264) |
| Branch Coverage | 50.23% (133073/264941) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Keep projected structs readable with nested Iceberg equality keys in V1 Parquet [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69547:
URL: https://github.com/apache/doris/pull/69547#issuecomment-6383866800
<details>
<summary>TPC-DS: <b>Total hot run time: 150363 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9858972766f2fcb75452e2e887507ae9e2c631b9, data reload: false
query5 4323 584 443 443
query6 424 204 186 186
query7 4826 438 249 249
query8 327 165 157 157
query9 8769 3114 3137 3114
query10 481 302 251 251
query11 5926 3868 3586 3586
query12 145 90 88 88
query13 1236 443 351 351
query14 6134 3917 3682 3682
query14_1 3428 3428 3429 3428
query15 212 196 182 182
query16 939 447 407 407
query17 867 649 548 548
query18 2423 441 311 311
query19 211 168 133 133
query20 84 81 84 81
query21 209 127 106 106
query22 12944 13036 12799 12799
query23 13317 13121 12358 12358
query23_1 12555 12490 12348 12348
query24 7385 839 467 467
query24_1 472 470 468 468
query25 541 424 380 380
query26 1254 252 134 134
query27 2775 446 271 271
query28 4637 1902 1922 1902
query29 1552 576 442 442
query30 291 222 182 182
query31 1024 930 803 803
query32 140 106 88 88
query33 513 289 222 222
query34 910 854 496 496
query35 742 753 633 633
query36 804 813 706 706
query37 124 97 83 83
query38 1788 1732 1695 1695
query39 789 790 744 744
query39_1 725 730 724 724
query40 223 114 92 92
query41 64 63 63 63
query42 86 84 84 84
query43 340 361 303 303
query44 1358 678 685 678
query45 175 192 164 164
query46 820 965 572 572
query47 2712 2879 2816 2816
query48 288 311 221 221
query49 569 391 288 288
query50 700 285 210 210
query51 10497 10420 10538 10420
query52 78 80 69 69
query53 183 206 152 152
query54 228 189 180 180
query55 69 67 62 62
query56 229 220 208 208
query57 1652 1615 1556 1556
query58 275 265 263 263
query59 2121 2170 1909 1909
query60 287 242 222 222
query61 145 144 143 143
query62 384 325 275 275
query63 186 154 153 153
query64 2764 915 725 725
query65 3308 3286 3293 3286
query66 1768 424 294 294
query67 19986 19999 19880 19880
query68 2925 1018 604 604
query69 370 279 235 235
query70 844 763 768 763
query71 290 224 209 209
query72 2664 2486 2258 2258
query73 516 529 294 294
query74 4639 4551 4350 4350
query75 2322 2283 1902 1902
query76 2314 1023 611 611
query77 354 373 267 267
query78 9620 9723 9169 9169
query79 889 881 542 542
query80 537 448 364 364
query81 528 324 274 274
query82 201 133 107 107
query83 310 209 209 209
query84 313 132 107 107
query85 907 469 434 434
query86 262 242 233 233
query87 1970 1995 1802 1802
query88 3652 2735 2638 2638
query89 319 277 239 239
query90 1971 166 163 163
query91 154 147 122 122
query92 92 88 84 84
query93 1006 997 577 577
query94 455 304 272 272
query95 573 390 299 299
query96 671 504 236 236
query97 2428 2427 2321 2321
query98 161 157 146 146
query99 757 765 643 643
Total cold run time: 232360 ms
Total hot run time: 150363 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Keep projected structs readable with nested Iceberg equality keys in V1 Parquet [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69547:
URL: https://github.com/apache/doris/pull/69547#issuecomment-6383910821
<details>
<summary>ClickBench: <b>Total hot run time: 24.22 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9858972766f2fcb75452e2e887507ae9e2c631b9, data reload: false
query1 0.00 0.00 0.00
query2 0.09 0.05 0.05
query3 0.26 0.09 0.07
query4 1.61 0.09 0.09
query5 0.24 0.25 0.24
query6 1.16 0.68 0.69
query7 0.04 0.01 0.01
query8 0.05 0.04 0.03
query9 0.40 0.34 0.36
query10 0.59 0.58 0.61
query11 0.20 0.16 0.14
query12 0.18 0.15 0.14
query13 0.48 0.48 0.48
query14 1.03 1.01 1.02
query15 0.64 0.63 0.60
query16 0.36 0.36 0.35
query17 1.13 1.17 1.14
query18 0.20 0.19 0.20
query19 2.10 1.94 1.99
query20 0.02 0.01 0.02
query21 15.42 0.19 0.13
query22 4.96 0.06 0.05
query23 16.31 0.33 0.12
query24 2.77 1.01 0.45
query25 0.11 0.06 0.06
query26 1.40 0.21 0.16
query27 0.05 0.06 0.05
query28 2.79 0.77 0.36
query29 12.50 4.18 3.27
query30 0.28 0.13 0.10
query31 2.76 0.55 0.32
query32 3.23 0.60 0.49
query33 3.16 3.25 3.12
query34 15.36 3.97 3.36
query35 3.35 3.32 3.31
query36 0.55 0.44 0.43
query37 0.09 0.07 0.06
query38 0.05 0.04 0.03
query39 0.04 0.03 0.03
query40 0.17 0.14 0.15
query41 0.09 0.03 0.03
query42 0.04 0.03 0.03
query43 0.05 0.03 0.04
Total cold run time: 96.31 s
Total hot run time: 24.22 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Keep projected structs readable with nested Iceberg equality keys in V1 Parquet [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69547:
URL: https://github.com/apache/doris/pull/69547#issuecomment-6378036670
<details>
<summary>ClickBench: <b>Total hot run time: 24.22 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5acba3e2c976c12a28fb9098cbcd08b1b697669d, data reload: false
query1 0.00 0.00 0.01
query2 0.09 0.05 0.05
query3 0.26 0.10 0.09
query4 1.61 0.10 0.10
query5 0.23 0.24 0.24
query6 1.17 0.68 0.67
query7 0.03 0.00 0.00
query8 0.06 0.04 0.04
query9 0.40 0.34 0.32
query10 0.57 0.61 0.58
query11 0.20 0.15 0.14
query12 0.19 0.15 0.15
query13 0.50 0.49 0.50
query14 1.02 1.03 1.01
query15 0.64 0.62 0.61
query16 0.34 0.33 0.34
query17 1.13 1.12 1.12
query18 0.23 0.19 0.20
query19 2.12 2.02 2.00
query20 0.01 0.02 0.01
query21 15.44 0.21 0.13
query22 4.94 0.05 0.07
query23 16.23 0.33 0.13
query24 3.22 0.46 0.32
query25 0.11 0.06 0.04
query26 0.87 0.22 0.15
query27 0.06 0.03 0.04
query28 2.64 0.75 0.36
query29 12.47 4.11 3.27
query30 0.28 0.12 0.10
query31 2.77 0.55 0.32
query32 3.23 0.59 0.51
query33 3.24 3.24 3.28
query34 15.32 3.97 3.35
query35 3.38 3.34 3.35
query36 0.54 0.42 0.41
query37 0.10 0.06 0.07
query38 0.04 0.03 0.04
query39 0.03 0.03 0.03
query40 0.17 0.15 0.14
query41 0.08 0.03 0.02
query42 0.04 0.03 0.02
query43 0.05 0.03 0.03
Total cold run time: 96.05 s
Total hot run time: 24.22 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Keep projected structs readable with nested Iceberg equality keys in V1 Parquet [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69547:
URL: https://github.com/apache/doris/pull/69547#issuecomment-6378521234
# BE UT Coverage Report
Increment line coverage `92.00% (184/200)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/5acba3e2c976c12a28fb9098cbcd08b1b697669d_5acba3e2c976c12a28fb9098cbcd08b1b697669d/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/5acba3e2c976c12a28fb9098cbcd08b1b697669d_5acba3e2c976c12a28fb9098cbcd08b1b697669d/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 66.59% (31999/48051) |
| Line Coverage | 51.43% (337882/656983) |
| Region Coverage | 46.79% (273420/584381) |
| Branch Coverage | 48.54% (128925/265599) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Keep projected structs readable with nested Iceberg equality keys in V1 Parquet [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69547:
URL: https://github.com/apache/doris/pull/69547#issuecomment-6368675366
<details>
<summary>ClickBench: <b>Total hot run time: 24.49 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6e2e3711b375b242ad91f8bce250e8d94e93a3e5, data reload: false
query1 0.00 0.00 0.00
query2 0.09 0.06 0.05
query3 0.26 0.08 0.08
query4 1.61 0.09 0.11
query5 0.23 0.25 0.24
query6 1.17 0.68 0.68
query7 0.04 0.01 0.01
query8 0.05 0.04 0.04
query9 0.40 0.35 0.35
query10 0.58 0.64 0.61
query11 0.20 0.15 0.14
query12 0.18 0.14 0.15
query13 0.48 0.48 0.49
query14 1.04 1.02 1.02
query15 0.64 0.61 0.62
query16 0.35 0.34 0.35
query17 1.13 1.10 1.10
query18 0.21 0.20 0.20
query19 2.04 1.98 2.07
query20 0.02 0.01 0.01
query21 15.48 0.21 0.13
query22 4.88 0.06 0.05
query23 16.37 0.31 0.13
query24 3.18 0.42 0.48
query25 0.10 0.06 0.07
query26 1.00 0.22 0.15
query27 0.05 0.04 0.04
query28 3.46 0.73 0.38
query29 12.48 4.18 3.28
query30 0.28 0.13 0.10
query31 2.77 0.57 0.32
query32 3.22 0.59 0.51
query33 3.13 3.24 3.29
query34 15.35 3.99 3.39
query35 3.34 3.34 3.34
query36 0.54 0.44 0.43
query37 0.09 0.07 0.06
query38 0.06 0.04 0.03
query39 0.04 0.03 0.03
query40 0.17 0.15 0.14
query41 0.09 0.03 0.03
query42 0.04 0.03 0.03
query43 0.04 0.03 0.04
Total cold run time: 96.88 s
Total hot run time: 24.49 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org