You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "morningman (via GitHub)" <gi...@apache.org> on 2026/10/18 17:08:59 UTC
[PR] [refactor](row-binlog) Derive binlog column indexes from column names [doris]
morningman opened a new pull request, #69242:
URL: https://github.com/apache/doris/pull/69242
### What problem does this PR solve?
Issue Number: None
Related PR: #65810
Problem Summary:
**In short.** Row binlog tablets keep the positions of their three hidden columns (`__DORIS_BINLOG_TSO__`, `__DORIS_BINLOG_LSN__`, `__DORIS_BINLOG_OP__`) in `TabletSchema`. Since #65810 the FE no longer sends them, so every row binlog tablet created since then has -1 in its tablet meta. Nothing reads that -1 today, but any new reader would. Only the BE unit tests still fill these fields, and that takes them down a path production never runs. This PR makes the BE derive the three positions from the column names whenever a schema is loaded. It also retires the thrift and protobuf fields that used to carry them. Nothing that runs today changes.
**Background.**
- A row binlog tablet stores change events. Besides the user columns, its schema has three hidden columns, `__DORIS_BINLOG_TSO__`, `__DORIS_BINLOG_LSN__` and `__DORIS_BINLOG_OP__`. `TabletSchema` keeps their positions in `_binlog_{tso,lsn,op}_col_idx`.
- The positions came from thrift `TTabletSchema` 28-30 when a tablet was created, and were persisted in `TabletSchemaPB` 41-43 (`TabletSchemaCloudPB` 42-44).
- Schemas built column by column already derive them from the names: `TabletSchema::append_column` and `build_current_tablet_schema`. Only `TabletSchema::init_from_pb`, which loads a schema from its PB, took the stored values.
**The problem.**
- #65810 stopped the FE from setting `TTabletSchema` 28-30 (`CreateReplicaTask`, and `CloudInternalCatalog` for cloud). Every row binlog tablet created since then has -1 in:
- its tablet meta schema;
- the meta service schema entry, in cloud mode;
- every schema loaded from those, such as cloud rowsets synced from the meta service and compaction outputs.
- Nothing reads these -1 today. Every reader gets a schema whose positions are derived from the names: loads (`build_current_tablet_schema`), scans and the publish-phase writer (`append_column`), and the rowset written by the same load. Still:
1. Any new code that reads the positions from the tablet schema gets -1.
2. If a load ever came with an index schema without column unique ids, `RowsetBuilder` would fall back to copying the tablet schema, and row binlog derivation would hit `CHECK(idx >= 0)`.
3. The BE unit tests (`create_row_binlog_tablet_schema` and others) still set 28-30 or the PB fields by hand, so they exercise a path production never takes.
**How this PR fixes it.**
1. `TabletSchema::init_from_pb` derives the three positions with `field_index(<column name>)`, as `append_column` does. A schema now has the same positions whether it was built column by column or loaded from a PB.
2. The stored values are no longer needed, so the fields are retired:
| File | Message | Ids | Old fields |
|---|---|---|---|
| AgentService.thrift | `TTabletSchema` | 28, 29, 30 | `binlog_tso_idx`, `binlog_lsn_idx`, `binlog_op_idx` (commented out) |
| olap_file.proto | `TabletSchemaPB` | 41, 42, 43 | `binlog_tso_col_idx`, `binlog_lsn_col_idx`, `binlog_op_col_idx` (`reserved`) |
| olap_file.proto | `TabletSchemaCloudPB` | 42, 43, 44 | same (`reserved`) |
`TabletSchema::to_schema_pb`, `TabletMeta` and the cloud `pb_convert` stop writing and copying them.
3. The BE unit tests stop setting them. A new `TabletSchemaTest.test_binlog_col_idx_from_pb_column_names` checks the positions:
- right after `init_from_pb`;
- after a round trip through the PB;
- on a schema without these columns, where they stay -1.
**Compatibility.**
- None of these ids exists on branch-4.2, branch-4.1, branch-4.0 or branch-3.1.
- Meta written by an older master still loads. Protobuf keeps the reserved ids as unknown fields, and the positions are derived anyway.
- An older master BE that reads meta written by this one gets -1, as it already does for every row binlog tablet created since #65810.
- With the positions derived, `is_tso_enabled()` is now also true on a row binlog tablet's own schema. Two callers read it from the tablet schema, and neither changes behavior:
- `CloudTablet::try_make_committed_rs_visible` returned without doing anything for a row binlog tablet before, and it still does. Row binlog tablets are not merge-on-write and never put rowsets into the committed rowset manager.
- The cloud delete task never reaches a row binlog tablet, because predicate DELETE is rejected on these tables.
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [x] Regression test
- [x] Unit Test
- [x] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Build: `build.sh --fe --be` (BE Release; FE with checkstyle).
- Regression test: `row_binlog_p0` (23 suites) and `table_stream_p0` (16 suites) pass.
- Unit Test, BE (`run-be-ut.sh`), 191 tests in 19 suites, all pass:
- the new `TabletSchemaTest.test_binlog_col_idx_from_pb_column_names`;
- `TabletSchemaTest`, `TabletMetaTest`, `PbConvert`;
- every suite that builds a row binlog schema: `RowBinlogDeriveTest`, `RowBinlogVmergeCompactionTest`, `SegmentFlusherFormatTest`, `SegmentFlusherTransformFormatTest`, `GroupRowsetBuilderTest`, `GroupRowsetWriterTest`, `CloudGroupRowsetBuilderWriterTest`, `TestEngineStorageMigrationTask`;
- the merge-on-write transform suites: `HistoricalRowFetcherTest`, `KeyProbeTest`, `RowCacheProbeTest`, `HistoricalRowRetrieverTest`, `FixedPartialUpdateTest`, `FlexiblePartialUpdateTest`, `ValidateStageTest`, `VariantRowStoreTest`.
- Manual test, local mode:
1. Create a row binlog table and load it. The row binlog tablet meta (`/api/meta/header`) no longer has the three fields.
2. Restart the cluster and load again.
3. Run a binlog compaction (`[0-1]` ... `[5-5]` into `[0-5]`).
4. Read through `binlog()`: every event comes back.
- Behavior changed:
- [x] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [x] No.
- [ ] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](row-binlog) Derive binlog column indexes from column names [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #69242:
URL: https://github.com/apache/doris/pull/69242#issuecomment-6252736537
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](row-binlog) Derive binlog column indexes from column names [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69242:
URL: https://github.com/apache/doris/pull/69242#issuecomment-6253633606
<details>
<summary>ClickBench: <b>Total hot run time: 25.62 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 26d02dcf8ad215de779cc73ba7b60f65cc2055bf, data reload: false
query1 0.01 0.01 0.00
query2 0.15 0.09 0.08
query3 0.37 0.21 0.21
query4 1.61 0.20 0.20
query5 0.30 0.29 0.29
query6 1.17 0.67 0.65
query7 0.04 0.01 0.01
query8 0.09 0.07 0.07
query9 0.48 0.35 0.36
query10 0.59 0.58 0.60
query11 0.33 0.18 0.18
query12 0.31 0.20 0.20
query13 0.51 0.50 0.51
query14 0.95 0.97 0.94
query15 0.67 0.60 0.59
query16 0.37 0.36 0.37
query17 1.05 1.05 1.03
query18 0.31 0.27 0.28
query19 1.92 1.90 1.90
query20 0.02 0.02 0.01
query21 15.40 0.35 0.30
query22 4.78 0.14 0.12
query23 15.85 0.50 0.29
query24 2.56 0.55 0.39
query25 0.15 0.10 0.10
query26 0.72 0.26 0.21
query27 0.11 0.10 0.09
query28 3.38 0.86 0.42
query29 12.51 4.34 3.43
query30 0.37 0.23 0.23
query31 2.76 0.63 0.36
query32 3.23 0.63 0.50
query33 2.98 2.93 2.94
query34 15.81 4.07 3.45
query35 3.39 3.35 3.40
query36 0.58 0.51 0.49
query37 0.13 0.09 0.10
query38 0.07 0.07 0.06
query39 0.07 0.06 0.07
query40 0.18 0.17 0.16
query41 0.11 0.06 0.05
query42 0.06 0.07 0.06
query43 0.07 0.06 0.06
Total cold run time: 96.52 s
Total hot run time: 25.62 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](row-binlog) Derive binlog column indexes from column names [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #69242:
URL: https://github.com/apache/doris/pull/69242#discussion_r4307955641
##########
be/src/storage/tablet/tablet_schema.cpp:
##########
@@ -1463,9 +1464,6 @@ void TabletSchema::to_schema_pb(TabletSchemaPB* tablet_schema_pb) const {
tablet_schema_pb->set_skip_bitmap_col_idx(_skip_bitmap_col_idx);
tablet_schema_pb->set_commit_tso_col_idx(_commit_tso_col_idx);
tablet_schema_pb->set_row_lsn_col_idx(_row_lsn_col_idx);
- tablet_schema_pb->set_binlog_tso_col_idx(_binlog_tso_col_idx);
- tablet_schema_pb->set_binlog_lsn_col_idx(_binlog_lsn_col_idx);
- tablet_schema_pb->set_binlog_op_col_idx(_binlog_op_col_idx);
tablet_schema_pb->set_inverted_index_storage_format(_inverted_index_storage_format);
Review Comment:
[P2] Keep emitting binlog ordinals for older BE recovery
Dropping these writes also removes the indexes from committed rowset metadata, which had valid values even when the tablet's metadata contained `-1`. If a local BE commits a row-binlog partial update, then is rolled back to the pre-PR master before publish, `DataDir::load` restores the attached rowset's existing PB schema and `TxnManager` restores its partial-update info. That reader initializes the absent LSN ordinal to `-1`; when publish aligns against a conflicting same-key row, `BaseTablet::calc_segment_delete_bitmap` casts it to `UINT32_MAX` and `create_storage_block` indexes `_cols[cid]`, causing an out-of-bounds access/BE crash. Please keep the deprecated PB fields and serialization writes for older readers while deriving the values on read, and cover recovery of such a pending transaction across rollback.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](row-binlog) Derive binlog column indexes from column names [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69242:
URL: https://github.com/apache/doris/pull/69242#issuecomment-6252712496
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://doris.apache.org/community/how-to-contribute/review-and-ci).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](row-binlog) Derive binlog column indexes from column names [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69242:
URL: https://github.com/apache/doris/pull/69242#issuecomment-6254944131
# BE Regression && UT Coverage Report
Increment line coverage `100% (0/0)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69242_26d02dcf8ad215de779cc73ba7b60f65cc2055bf_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69242_26d02dcf8ad215de779cc73ba7b60f65cc2055bf_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 77.20% (35765/46328) |
| Line Coverage | 62.44% (403783/646635) |
| Region Coverage | 58.84% (340685/578968) |
| Branch Coverage | 59.83% (157531/263285) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](row-binlog) Derive binlog column indexes from column names [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69242:
URL: https://github.com/apache/doris/pull/69242#issuecomment-6269452345
# FE UT Coverage Report
Increment line coverage `` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69242_a2d935313ee0dbf83277e32e854d8b28f028209d/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69242_a2d935313ee0dbf83277e32e854d8b28f028209d/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](row-binlog) Derive binlog column indexes from column names [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69242:
URL: https://github.com/apache/doris/pull/69242#issuecomment-6268309190
# Cloud UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/a2d935313ee0dbf83277e32e854d8b28f028209d_a2d935313ee0dbf83277e32e854d8b28f028209d_cloud/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/a2d935313ee0dbf83277e32e854d8b28f028209d_a2d935313ee0dbf83277e32e854d8b28f028209d_cloud/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 77.98% (2079/2666) |
| Line Coverage | 66.20% (38266/57806) |
| Region Coverage | 53.61% (35866/66898) |
| Branch Coverage | 57.07% (11543/20225) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](row-binlog) Derive binlog column indexes from column names [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69242:
URL: https://github.com/apache/doris/pull/69242#issuecomment-6253693046
# FE UT Coverage Report
Increment line coverage `` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69242_26d02dcf8ad215de779cc73ba7b60f65cc2055bf/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69242_26d02dcf8ad215de779cc73ba7b60f65cc2055bf/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](row-binlog) Derive binlog column indexes from column names [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #69242:
URL: https://github.com/apache/doris/pull/69242#issuecomment-6252713042
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](row-binlog) Derive binlog column indexes from column names [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69242:
URL: https://github.com/apache/doris/pull/69242#issuecomment-6253483940
<details>
<summary>TPC-H: <b>Total hot run time: 29798 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 26d02dcf8ad215de779cc73ba7b60f65cc2055bf, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17586 5641 5574 5574
q2 2079 321 287 287
q3 10220 1405 864 864
q4 4712 472 337 337
q5 7625 802 540 540
q6 197 193 159 159
q7 784 811 612 612
q8 10335 1409 1078 1078
q9 5748 4341 4290 4290
q10 6872 1305 1018 1018
q11 436 255 241 241
q12 682 413 293 293
q13 18157 3573 2995 2995
q14 275 275 248 248
q15 q16 752 731 667 667
q17 1228 750 600 600
q18 7137 6238 6222 6222
q19 1134 960 631 631
q20 424 355 227 227
q21 5749 2600 2597 2597
q22 392 363 318 318
Total cold run time: 102524 ms
Total hot run time: 29798 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 6555 6509 6469 6469
q2 753 624 597 597
q3 5601 5684 5523 5523
q4 2237 2228 1601 1601
q5 5792 5737 5767 5737
q6 243 197 154 154
q7 2247 2007 1737 1737
q8 3197 2686 2584 2584
q9 7665 7568 7543 7543
q10 3685 3599 3254 3254
q11 612 424 374 374
q12 661 703 504 504
q13 3333 3573 3065 3065
q14 293 286 262 262
q15 q16 666 681 626 626
q17 7887 7072 6941 6941
q18 13195 12263 13159 12263
q19 952 886 875 875
q20 2198 2165 1919 1919
q21 5956 4795 4963 4795
q22 513 464 412 412
Total cold run time: 74241 ms
Total hot run time: 67235 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](row-binlog) Derive binlog column indexes from column names [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman merged PR #69242:
URL: https://github.com/apache/doris/pull/69242
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](row-binlog) Derive binlog column indexes from column names [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69242:
URL: https://github.com/apache/doris/pull/69242#issuecomment-6270548537
<details>
<summary>TPC-DS: <b>Total hot run time: 152022 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a2d935313ee0dbf83277e32e854d8b28f028209d, data reload: false
query5 4349 585 455 455
query6 439 223 207 207
query7 4818 447 227 227
query8 327 172 155 155
query9 8798 3171 3181 3171
query10 460 314 264 264
query11 5967 3925 3613 3613
query12 152 107 94 94
query13 1304 481 331 331
query14 6171 3979 3721 3721
query14_1 3524 3465 3455 3455
query15 215 208 181 181
query16 929 378 398 378
query17 822 645 516 516
query18 2401 417 303 303
query19 199 161 129 129
query20 87 85 82 82
query21 219 129 110 110
query22 13144 13166 12970 12970
query23 13508 12917 12503 12503
query23_1 12613 12656 12514 12514
query24 7353 816 457 457
query24_1 466 468 470 468
query25 516 403 337 337
query26 1255 253 135 135
query27 2785 444 272 272
query28 4596 1957 1954 1954
query29 1544 579 438 438
query30 299 212 185 185
query31 1042 946 821 821
query32 148 94 95 94
query33 528 316 254 254
query34 904 878 515 515
query35 742 759 642 642
query36 811 806 723 723
query37 133 100 86 86
query38 1780 1738 1683 1683
query39 794 813 754 754
query39_1 744 731 745 731
query40 228 121 101 101
query41 72 68 68 68
query42 91 88 87 87
query43 350 365 322 322
query44 1365 703 705 703
query45 187 186 173 173
query46 866 981 571 571
query47 3054 3012 2886 2886
query48 314 313 231 231
query49 586 398 309 309
query50 713 288 219 219
query51 10756 10323 10752 10323
query52 83 84 78 78
query53 197 224 157 157
query54 261 206 198 198
query55 80 72 67 67
query56 240 234 231 231
query57 1669 1637 1569 1569
query58 303 277 263 263
query59 2104 2152 1967 1967
query60 304 255 235 235
query61 170 170 168 168
query62 397 333 289 289
query63 198 166 169 166
query64 2887 1084 766 766
query65 3438 3321 3289 3289
query66 1769 418 306 306
query67 20133 20363 20249 20249
query68 2968 1017 615 615
query69 379 290 247 247
query70 854 776 790 776
query71 302 241 214 214
query72 2821 2621 2296 2296
query73 524 545 299 299
query74 4668 4543 4380 4380
query75 2337 2289 1944 1944
query76 2312 1040 629 629
query77 352 388 283 283
query78 9783 9749 9187 9187
query79 993 866 521 521
query80 1263 456 349 349
query81 576 323 281 281
query82 562 141 106 106
query83 306 204 183 183
query84 304 123 107 107
query85 867 467 372 372
query86 393 233 233 233
query87 2030 1955 1811 1811
query88 3770 2700 2674 2674
query89 329 284 240 240
query90 1865 174 170 170
query91 163 159 126 126
query92 95 87 92 87
query93 1162 998 582 582
query94 650 312 268 268
query95 577 389 297 297
query96 687 495 236 236
query97 2442 2442 2288 2288
query98 163 153 148 148
query99 781 764 636 636
Total cold run time: 236788 ms
Total hot run time: 152022 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](row-binlog) Derive binlog column indexes from column names [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69242:
URL: https://github.com/apache/doris/pull/69242#issuecomment-6254872383
# FE Regression Coverage Report
Increment line coverage ` 100% (0/0)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69242_26d02dcf8ad215de779cc73ba7b60f65cc2055bf_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69242_26d02dcf8ad215de779cc73ba7b60f65cc2055bf_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](row-binlog) Derive binlog column indexes from column names [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69242:
URL: https://github.com/apache/doris/pull/69242#issuecomment-6270342248
<details>
<summary>TPC-H: <b>Total hot run time: 28672 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a2d935313ee0dbf83277e32e854d8b28f028209d, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17605 5237 5099 5099
q2 2108 290 255 255
q3 10446 1395 828 828
q4 4723 465 336 336
q5 7824 798 530 530
q6 225 188 151 151
q7 768 786 607 607
q8 10123 1373 1074 1074
q9 5593 4245 4201 4201
q10 6896 1306 1016 1016
q11 438 251 223 223
q12 698 415 290 290
q13 18163 3674 3081 3081
q14 280 265 237 237
q15 q16 725 716 670 670
q17 1258 751 607 607
q18 7048 6203 6125 6125
q19 1126 961 622 622
q20 400 349 232 232
q21 5581 2219 2327 2219
q22 357 299 269 269
Total cold run time: 102385 ms
Total hot run time: 28672 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 5637 5624 5647 5624
q2 691 570 566 566
q3 5220 5410 4924 4924
q4 2042 2158 1570 1570
q5 5290 5219 6016 5219
q6 252 191 165 165
q7 2288 2069 1807 1807
q8 3110 2711 2728 2711
q9 8057 7907 7877 7877
q10 3719 3986 3265 3265
q11 662 463 412 412
q12 692 708 536 536
q13 3656 3838 3232 3232
q14 324 334 321 321
q15 q16 698 723 657 657
q17 8316 7308 6971 6971
q18 13018 12246 12923 12246
q19 947 811 798 798
q20 2148 2122 1916 1916
q21 5682 4697 4709 4697
q22 492 457 418 418
Total cold run time: 72941 ms
Total hot run time: 65932 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](row-binlog) Derive binlog column indexes from column names [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #69242:
URL: https://github.com/apache/doris/pull/69242#issuecomment-6264992958
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](row-binlog) Derive binlog column indexes from column names [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #69242:
URL: https://github.com/apache/doris/pull/69242#issuecomment-6264994374
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](row-binlog) Derive binlog column indexes from column names [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69242:
URL: https://github.com/apache/doris/pull/69242#issuecomment-6271913627
# FE Regression Coverage Report
Increment line coverage ` 100% (0/0)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69242_a2d935313ee0dbf83277e32e854d8b28f028209d_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69242_a2d935313ee0dbf83277e32e854d8b28f028209d_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](row-binlog) Derive binlog column indexes from column names [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69242:
URL: https://github.com/apache/doris/pull/69242#issuecomment-6254905949
# BE UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/26d02dcf8ad215de779cc73ba7b60f65cc2055bf_26d02dcf8ad215de779cc73ba7b60f65cc2055bf/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/26d02dcf8ad215de779cc73ba7b60f65cc2055bf_26d02dcf8ad215de779cc73ba7b60f65cc2055bf/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 66.37% (31729/47807) |
| Line Coverage | 51.12% (334147/653620) |
| Region Coverage | 46.46% (269977/581058) |
| Branch Coverage | 48.20% (127205/263926) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](row-binlog) Derive binlog column indexes from column names [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69242:
URL: https://github.com/apache/doris/pull/69242#issuecomment-6270657129
<details>
<summary>ClickBench: <b>Total hot run time: 25.38 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a2d935313ee0dbf83277e32e854d8b28f028209d, data reload: false
query1 0.00 0.00 0.00
query2 0.14 0.09 0.09
query3 0.34 0.13 0.13
query4 1.60 0.14 0.14
query5 0.26 0.26 0.25
query6 1.16 0.65 0.65
query7 0.04 0.01 0.00
query8 0.09 0.07 0.08
query9 0.45 0.36 0.37
query10 0.61 0.62 0.59
query11 0.33 0.18 0.18
query12 0.31 0.20 0.18
query13 0.51 0.53 0.52
query14 0.94 0.95 0.94
query15 0.67 0.58 0.59
query16 0.37 0.37 0.36
query17 1.06 1.05 1.01
query18 0.30 0.29 0.30
query19 1.97 1.96 1.87
query20 0.02 0.01 0.02
query21 15.43 0.34 0.30
query22 4.93 0.13 0.14
query23 15.85 0.49 0.30
query24 2.36 0.53 0.38
query25 0.15 0.10 0.10
query26 0.77 0.26 0.20
query27 0.10 0.10 0.09
query28 3.36 0.82 0.38
query29 12.57 4.38 3.42
query30 0.36 0.16 0.15
query31 2.76 0.58 0.32
query32 3.22 0.60 0.48
query33 2.96 3.04 3.13
query34 15.91 4.12 3.47
query35 3.39 3.38 3.39
query36 0.59 0.53 0.48
query37 0.12 0.10 0.09
query38 0.08 0.07 0.07
query39 0.05 0.06 0.05
query40 0.22 0.17 0.16
query41 0.11 0.05 0.06
query42 0.06 0.06 0.06
query43 0.07 0.06 0.06
Total cold run time: 96.59 s
Total hot run time: 25.38 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](row-binlog) Derive binlog column indexes from column names [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69242:
URL: https://github.com/apache/doris/pull/69242#issuecomment-6253582532
<details>
<summary>TPC-DS: <b>Total hot run time: 152278 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 26d02dcf8ad215de779cc73ba7b60f65cc2055bf, data reload: false
query5 4315 577 457 457
query6 419 214 212 212
query7 4799 446 226 226
query8 323 169 167 167
query9 8758 3991 3992 3991
query10 504 316 267 267
query11 6078 3802 3537 3537
query12 149 93 91 91
query13 1257 450 338 338
query14 6064 4024 3751 3751
query14_1 3541 3497 3513 3497
query15 209 200 182 182
query16 948 480 456 456
query17 857 659 551 551
query18 2430 435 316 316
query19 210 170 131 131
query20 85 96 83 83
query21 218 143 113 113
query22 12999 13018 12774 12774
query23 13515 12953 12498 12498
query23_1 12599 12487 12710 12487
query24 7091 848 471 471
query24_1 481 472 473 472
query25 539 419 368 368
query26 1266 259 140 140
query27 2763 444 277 277
query28 4596 1943 1955 1943
query29 1592 617 480 480
query30 297 217 186 186
query31 1006 924 815 815
query32 144 98 95 95
query33 515 314 250 250
query34 876 869 519 519
query35 742 775 645 645
query36 823 819 709 709
query37 132 100 89 89
query38 1834 1742 1688 1688
query39 793 796 760 760
query39_1 736 732 737 732
query40 218 123 98 98
query41 79 69 70 69
query42 88 85 86 85
query43 373 392 330 330
query44 1338 704 754 704
query45 179 179 159 159
query46 838 966 564 564
query47 3019 3004 2900 2900
query48 309 312 211 211
query49 562 385 290 290
query50 689 268 212 212
query51 10232 10279 10457 10279
query52 74 78 68 68
query53 186 205 150 150
query54 249 194 173 173
query55 75 64 62 62
query56 222 246 218 218
query57 1606 1588 1554 1554
query58 279 249 253 249
query59 2089 2114 1943 1943
query60 283 240 210 210
query61 146 142 134 134
query62 395 336 287 287
query63 198 160 169 160
query64 2761 916 780 780
query65 3295 3237 3252 3237
query66 1757 415 312 312
query67 20148 20291 20189 20189
query68 2987 1002 613 613
query69 379 284 247 247
query70 862 788 798 788
query71 285 220 210 210
query72 2745 2310 2248 2248
query73 546 533 306 306
query74 4609 4477 4291 4291
query75 2358 2308 1936 1936
query76 2294 1022 633 633
query77 344 351 288 288
query78 9791 9635 9152 9152
query79 1127 874 524 524
query80 1231 414 327 327
query81 585 321 266 266
query82 559 129 105 105
query83 295 200 180 180
query84 305 125 100 100
query85 845 430 387 387
query86 404 237 229 229
query87 2012 1958 1823 1823
query88 3666 2732 2701 2701
query89 344 298 251 251
query90 1869 181 176 176
query91 159 149 125 125
query92 96 87 97 87
query93 1150 1040 572 572
query94 654 302 273 273
query95 578 396 324 324
query96 655 534 233 233
query97 2461 2454 2346 2346
query98 159 151 148 148
query99 781 775 658 658
Total cold run time: 234856 ms
Total hot run time: 152278 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](row-binlog) Derive binlog column indexes from column names [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69242:
URL: https://github.com/apache/doris/pull/69242#issuecomment-6253161930
# Cloud UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/26d02dcf8ad215de779cc73ba7b60f65cc2055bf_26d02dcf8ad215de779cc73ba7b60f65cc2055bf_cloud/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/26d02dcf8ad215de779cc73ba7b60f65cc2055bf_26d02dcf8ad215de779cc73ba7b60f65cc2055bf_cloud/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 77.98% (2079/2666) |
| Line Coverage | 66.11% (38217/57806) |
| Region Coverage | 53.54% (35814/66898) |
| Branch Coverage | 57.05% (11538/20225) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org