You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "heguanhui (via GitHub)" <gi...@apache.org> on 2026/05/28 17:34:53 UTC
[PR] [feature](log) Add label and txn_id columns to audit_log for transactional SQL statements [doris]
heguanhui opened a new pull request, #63865:
URL: https://github.com/apache/doris/pull/63865
## What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary: The `audit_log` internal table did not record transaction information (`label` and `txn_id`) for transactional SQL statements (INSERT/UPDATE/DELETE/MERGE_INTO). Users and DBAs could not trace which transaction a particular DML operation belonged to, making it difficult to correlate audit log entries with transaction metadata. Additionally, the old-style DELETE path (via `DeleteHandler`) did not call `setOrUpdateInsertResult()`, causing `label`/`txn_id` to be missing even if they were recorded.
## What is changed and how it works?
1. **Add `label` and `txn_id` fields to `AuditEvent`** — Two new `@AuditField` annotated public fields with corresponding builder setters.
2. **Populate label/txn_id in `AuditLogHelper.logAuditLogImpl()`** — Only for transactional statement types (INSERT/UPDATE/DELETE/MERGE_INTO), read from `ConnectContext.insertResult`. Non-transactional statements (e.g., SELECT) get empty label and -1 txn_id.
3. **Add `label` and `txn_id` columns to `InternalSchema.AUDIT_SCHEMA`** — `label` as `varchar(128)`, `txn_id` as `bigint`. Existing audit_log tables will be auto-altered by `InternalSchemaInitializer.alterAuditSchemaIfNeeded()`.
4. **Serialize label/txn_id in `AuditLoader.fillLogBuffer()`** — Append the two new fields to the CSV buffer for stream load into audit_log.
5. **Remove duplicate `label` field from `LoadAuditEvent` and `StreamLoadAuditEvent`** — These subclasses now inherit `label` from the base `AuditEvent`, avoiding `Class.getFields()` reflection returning duplicate fields.
6. **Fix old-style DELETE path in `DeleteHandler.process()`** — Call `ctx.setOrUpdateInsertResult()` after successful commit so that label/txn_id are available for audit logging.
## Release note
Add `label` (varchar(128)) and `txn_id` (bigint) columns to the `audit_log` internal table. These columns record the transaction label and transaction ID for INSERT, UPDATE, DELETE, and MERGE INTO statements, enabling better transaction tracing and audit analysis.
## Check List (For Author)
- Test: Regression test / Unit Test
- Regression test: `test_audit_log_label_txn_id.groovy` (INSERT with/without label, UPDATE, DELETE, SELECT)
- Unit Test: `AuditLogHelperTest` (4 tests), `AuditLogBuilderTest` (8 tests), `AuditEventProcessorTest`, `InternalSchemaInitializerTest`, `InternalSchemaAlterTest`, `AuditLogWorkloadGroupTest` (20 tests total, all passed)
- Behavior changed: Yes — audit_log table now has two additional columns; old-style DELETE now populates insertResult
- Does this need documentation: Yes (will add separately)
## Check List (For Reviewer)
- [ ] I have added test cases for this bug fix or new feature
- [ ] This PR will not cause performance regression
- [ ] This PR will not break existing features
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](log) Add label and txn_id columns to audit_log for transactional SQL statements [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63865:
URL: https://github.com/apache/doris/pull/63865#issuecomment-4595969403
# FE UT Coverage Report
Increment line coverage ` 59.62% (31/52)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63865_d62e750d2dcd7dca12aa56d2ca69e36c07e7b5c3/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63865_d62e750d2dcd7dca12aa56d2ca69e36c07e7b5c3/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](log) Add label and txn_id columns to audit_log for transactional SQL statements [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63865:
URL: https://github.com/apache/doris/pull/63865#issuecomment-4567093625
<details>
<summary>TPC-DS: <b>Total hot run time: 171053 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 32708331fac5f1d28ec4781cc4e9a16a5f8e3014, data reload: false
query5 4326 645 512 512
query6 340 216 199 199
query7 4224 524 313 313
query8 367 243 227 227
query9 8802 4001 3977 3977
query10 448 334 300 300
query11 5751 2402 2189 2189
query12 177 127 120 120
query13 1302 573 430 430
query14 6096 5442 5133 5133
query14_1 4427 4473 4438 4438
query15 211 206 184 184
query16 1005 454 418 418
query17 1113 736 581 581
query18 2430 479 360 360
query19 217 208 169 169
query20 134 128 128 128
query21 215 144 115 115
query22 13615 13569 13458 13458
query23 17287 16625 16270 16270
query23_1 16309 16362 16490 16362
query24 7573 1760 1307 1307
query24_1 1315 1299 1317 1299
query25 592 513 443 443
query26 1314 320 175 175
query27 2731 588 349 349
query28 4470 2021 2005 2005
query29 990 673 520 520
query30 307 247 203 203
query31 1131 1071 976 976
query32 91 76 75 75
query33 559 355 306 306
query34 1190 1116 660 660
query35 775 794 699 699
query36 1416 1427 1225 1225
query37 157 107 92 92
query38 3230 3181 3075 3075
query39 919 911 902 902
query39_1 904 886 862 862
query40 236 147 127 127
query41 71 73 68 68
query42 110 110 122 110
query43 326 334 284 284
query44
query45 219 207 199 199
query46 1074 1182 719 719
query47 2385 2384 2264 2264
query48 398 410 313 313
query49 656 510 430 430
query50 962 369 255 255
query51 4336 4307 4395 4307
query52 106 108 93 93
query53 257 276 207 207
query54 331 283 271 271
query55 95 93 87 87
query56 313 319 310 310
query57 1427 1410 1334 1334
query58 313 275 270 270
query59 1599 1696 1425 1425
query60 320 320 294 294
query61 159 155 153 153
query62 704 647 585 585
query63 238 192 206 192
query64 2388 809 649 649
query65
query66 1707 489 350 350
query67 29986 29725 29519 29519
query68
query69 440 341 299 299
query70 1018 989 1004 989
query71 294 271 263 263
query72 3062 2705 2385 2385
query73 822 734 409 409
query74 5146 4943 4770 4770
query75 2698 2607 2292 2292
query76 2302 1137 755 755
query77 410 417 330 330
query78 12471 12423 11922 11922
query79 1441 1014 744 744
query80 1378 543 451 451
query81 489 280 238 238
query82 1352 161 121 121
query83 337 274 246 246
query84 262 145 112 112
query85 928 537 451 451
query86 449 352 311 311
query87 3439 3359 3192 3192
query88 3589 2692 2694 2692
query89 467 382 346 346
query90 1849 178 180 178
query91 179 178 163 163
query92 75 78 75 75
query93 1511 1467 892 892
query94 690 365 274 274
query95 688 462 339 339
query96 1007 781 357 357
query97 2732 2755 2645 2645
query98 236 224 224 224
query99 1222 1169 1028 1028
Total cold run time: 255125 ms
Total hot run time: 171053 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](log) Add label and txn_id columns to audit_log for transactional SQL statements [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #63865:
URL: https://github.com/apache/doris/pull/63865#discussion_r3332258586
##########
regression-test/suites/audit/test_audit_log_label_txn_id.groovy:
##########
@@ -0,0 +1,134 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_audit_log_label_txn_id","nonConcurrent") {
+ try {
Review Comment:
This test enables the global audit plugin but only disables it at the very end. Any failure between this line and the final `set global enable_audit_plugin = false` leaves the global setting enabled for later suites in the same regression run, which can make unrelated tests non-deterministic and keep writing audit rows. Please restore the original/global value in a `finally` block (or otherwise guarantee cleanup) around the body after the privilege check succeeds.
##########
fe/fe-core/src/main/java/org/apache/doris/qe/AuditLogHelper.java:
##########
@@ -69,12 +69,18 @@ public class AuditLogHelper {
private static final Logger LOG = LogManager.getLogger(AuditLogHelper.class);
private static final Set<String> LOG_PLAN_INFO_TYPES = Sets.newTreeSet(String.CASE_INSENSITIVE_ORDER);
+ private static final Set<String> TXN_STMT_TYPES = Sets.newTreeSet(String.CASE_INSENSITIVE_ORDER);
static {
Review Comment:
`ConnectContext.getInsertResult()` is the session's last insert result used by `SHOW LAST INSERT`, and it is not cleared at statement start. That means an audited transactional statement can inherit stale metadata from a previous successful insert/update/delete in the same connection. For example: run a successful `insert` so `insertResult` is populated, then run an `insert` that fails during analysis/execution before `setOrUpdateInsertResult()` is called, or a `delete` that reaches `processEmptyRelation()` and reports txnId `-1` without updating the context. `stmtType` is still `INSERT`/`DELETE`, so this block writes the previous statement's label/txn_id into the current audit row. The audit fields need to be scoped to the current statement/result, or `insertResult` must be cleared/validated for the current statement before it is used here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](log) Add label and txn_id columns to audit_log for transactional SQL statements [doris]
Posted by "heguanhui (via GitHub)" <gi...@apache.org>.
heguanhui commented on code in PR #63865:
URL: https://github.com/apache/doris/pull/63865#discussion_r3332244141
##########
fe/fe-core/src/main/java/org/apache/doris/catalog/InternalSchema.java:
##########
@@ -230,6 +230,11 @@ public class InternalSchema {
ScalarType.createType(PrimitiveType.STRING), ColumnNullableType.NULLABLE));
AUDIT_SCHEMA.add(new ColumnDef("compute_group",
ScalarType.createType(PrimitiveType.STRING), ColumnNullableType.NULLABLE));
+ // transaction info
+ AUDIT_SCHEMA.add(new ColumnDef("label",
+ ScalarType.createVarchar(Config.label_regex_length), ColumnNullableType.NULLABLE));
Review Comment:
Your proposal is very good. I will refer to your suggestion and adjust the code accordingly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](log) Add label and txn_id columns to audit_log for transactional SQL statements [doris]
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow commented on PR #63865:
URL: https://github.com/apache/doris/pull/63865#issuecomment-4608971457
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](log) Add label and txn_id columns to audit_log for transactional SQL statements [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63865:
URL: https://github.com/apache/doris/pull/63865#issuecomment-4595615899
<details>
<summary>TPC-H: <b>Total hot run time: 29155 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d62e750d2dcd7dca12aa56d2ca69e36c07e7b5c3, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17620 4249 4212 4212
q2 q3 10778 1418 841 841
q4 4687 477 348 348
q5 7572 888 580 580
q6 192 183 141 141
q7 835 839 650 650
q8 9350 1603 1716 1603
q9 6225 4512 4488 4488
q10 6869 1839 1520 1520
q11 451 280 259 259
q12 646 436 303 303
q13 18122 3393 2757 2757
q14 272 258 243 243
q15 q16 828 768 706 706
q17 1016 989 996 989
q18 6902 5691 5434 5434
q19 1309 1322 1108 1108
q20 512 415 261 261
q21 5902 2671 2403 2403
q22 441 369 309 309
Total cold run time: 100529 ms
Total hot run time: 29155 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4559 4479 4503 4479
q2 q3 4550 4987 4354 4354
q4 2156 2261 1415 1415
q5 4562 4443 4452 4443
q6 250 185 135 135
q7 1801 2242 1795 1795
q8 2774 2357 2277 2277
q9 8103 8056 8081 8056
q10 4790 4780 4334 4334
q11 612 465 409 409
q12 767 894 639 639
q13 3299 3687 2989 2989
q14 306 332 303 303
q15 q16 786 726 634 634
q17 1392 1363 1345 1345
q18 8299 7506 7254 7254
q19 1190 1092 1121 1092
q20 2226 2245 1967 1967
q21 5408 4767 4750 4750
q22 545 463 413 413
Total cold run time: 58375 ms
Total hot run time: 53083 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](log) Add label and txn_id columns to audit_log for transactional SQL statements [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63865:
URL: https://github.com/apache/doris/pull/63865#issuecomment-4609426521
# FE Regression Coverage Report
Increment line coverage ` 3.98% (47/1181)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63865_d62e750d2dcd7dca12aa56d2ca69e36c07e7b5c3_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63865_d62e750d2dcd7dca12aa56d2ca69e36c07e7b5c3_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](log) Add label and txn_id columns to audit_log for transactional SQL statements [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63865:
URL: https://github.com/apache/doris/pull/63865#issuecomment-4568062105
# FE Regression Coverage Report
Increment line coverage ` 96.15% (25/26)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63865_32708331fac5f1d28ec4781cc4e9a16a5f8e3014_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63865_32708331fac5f1d28ec4781cc4e9a16a5f8e3014_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](log) Add label and txn_id columns to audit_log for transactional SQL statements [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63865:
URL: https://github.com/apache/doris/pull/63865#issuecomment-4567012305
<details>
<summary>TPC-H: <b>Total hot run time: 31029 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 32708331fac5f1d28ec4781cc4e9a16a5f8e3014, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17820 3981 3920 3920
q2 q3 10812 1380 803 803
q4 4682 473 351 351
q5 7586 2237 2071 2071
q6 377 172 135 135
q7 949 791 626 626
q8 9541 1756 1563 1563
q9 7130 4964 4941 4941
q10 6443 2226 1836 1836
q11 445 266 236 236
q12 691 430 295 295
q13 18238 3349 2786 2786
q14 254 259 237 237
q15 q16 819 776 705 705
q17 996 912 851 851
q18 6917 5933 5578 5578
q19 1185 1195 1069 1069
q20 523 396 258 258
q21 5797 2563 2461 2461
q22 421 354 307 307
Total cold run time: 101626 ms
Total hot run time: 31029 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4331 4238 4259 4238
q2 q3 4550 4940 4304 4304
q4 2055 2202 1360 1360
q5 4393 4292 4702 4292
q6 257 203 142 142
q7 2067 1798 1551 1551
q8 2464 2100 2152 2100
q9 8009 8003 7940 7940
q10 4892 4713 4437 4437
q11 575 425 370 370
q12 771 749 528 528
q13 3251 3613 2941 2941
q14 302 303 276 276
q15 q16 710 727 632 632
q17 1343 1312 1334 1312
q18 7990 7465 6748 6748
q19 1129 1089 1082 1082
q20 2219 2223 1939 1939
q21 5200 4505 4395 4395
q22 516 446 439 439
Total cold run time: 57024 ms
Total hot run time: 51026 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](log) Add label and txn_id columns to audit_log for transactional SQL statements [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63865:
URL: https://github.com/apache/doris/pull/63865#issuecomment-4596650066
# FE Regression Coverage Report
Increment line coverage ` 90.38% (47/52)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63865_d62e750d2dcd7dca12aa56d2ca69e36c07e7b5c3_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63865_d62e750d2dcd7dca12aa56d2ca69e36c07e7b5c3_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](log) Add label and txn_id columns to audit_log for transactional SQL statements [doris]
Posted by "heguanhui (via GitHub)" <gi...@apache.org>.
heguanhui commented on PR #63865:
URL: https://github.com/apache/doris/pull/63865#issuecomment-4597848402
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](log) Add label and txn_id columns to audit_log for transactional SQL statements [doris]
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow commented on PR #63865:
URL: https://github.com/apache/doris/pull/63865#issuecomment-4591577708
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](log) Add label and txn_id columns to audit_log for transactional SQL statements [doris]
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow commented on PR #63865:
URL: https://github.com/apache/doris/pull/63865#issuecomment-4590319183
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](log) Add label and txn_id columns to audit_log for transactional SQL statements [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #63865:
URL: https://github.com/apache/doris/pull/63865#discussion_r3345877669
##########
fe/fe-core/src/main/java/org/apache/doris/qe/AuditLogHelper.java:
##########
@@ -279,6 +290,15 @@ private static void logAuditLogImpl(ConnectContext ctx, String origStmt, Stateme
.setCloudCluster(Strings.isNullOrEmpty(cluster) ? "UNKNOWN" : cluster)
.setWorkloadGroup(ctx.getWorkloadGroupName());
+ // load_label and txn_id for transactional statements (INSERT/UPDATE/DELETE)
Review Comment:
This only reads `insertResult` from the local `ConnectContext`, but normal `INSERT`/`UPDATE`/`DELETE`/`MERGE_INTO` on a non-master FE are forwarded because these commands implement `ForwardWithSync`. The master-side `proxyExecute()` runs the DML and populates the proxy context's insert result, but `TMasterOpResult` only returns packet/status/affected rows/txnLoadInfo, and `FEOpExecutor.execute()` never copies label/txnId back into the original context. The observer/follower then audits the statement here with empty `load_label` and `txn_id = -1` even though the DML succeeded. Please propagate the insert result through the forward response, or otherwise set these audit fields from the forwarded execution result before logging.
##########
fe/fe-core/src/main/java/org/apache/doris/plugin/audit/LoadAuditEvent.java:
##########
@@ -65,8 +63,8 @@ public AuditEventBuilder setJobId(long jobId) {
return this;
}
- public AuditEventBuilder setLabel(String label) {
- auditEvent.label = label;
+ public AuditEventBuilder setLoadLabel(String loadLabel) {
Review Comment:
`AuditLogBuilder.auditLoadLog()` still filters and emits `LOAD_ANNONATION_NAMES`, which contains `Label`, but this PR removed the only `@AuditField(value = "Label")` from `LoadAuditEvent` and stores the value in inherited `LoadLabel` instead. Because the builder map is keyed by annotation value, every `LOAD_SUCCEED` line in `fe.load.audit.log` will now contain `|Label=null` even when the load has a label. Please keep a `Label` audit field for load audit events or update the load audit builder/output name together with this rename.
##########
fe/fe-core/src/main/java/org/apache/doris/plugin/audit/StreamLoadAuditEvent.java:
##########
@@ -64,8 +62,8 @@ public AuditEventBuilder setEventType(EventType eventType) {
return this;
}
- public AuditEventBuilder setLabel(String label) {
- auditEvent.label = label;
+ public AuditEventBuilder setLoadLabel(String loadLabel) {
Review Comment:
The stream-load text audit path has the same regression: `AuditLogBuilder.auditStreamLoadLog()` still looks for annotation `Label`, but this class no longer exposes any `@AuditField(value = "Label")`. As a result `fe.stream_load.audit.log` will print `|Label=null` for completed stream loads. Please keep the legacy `Label` audit field here or update the stream-load audit builder/output contract consistently.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](log) Add label and txn_id columns to audit_log for transactional SQL statements [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63865:
URL: https://github.com/apache/doris/pull/63865#issuecomment-4595698450
<details>
<summary>TPC-DS: <b>Total hot run time: 171445 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d62e750d2dcd7dca12aa56d2ca69e36c07e7b5c3, data reload: false
query5 4320 689 524 524
query6 337 239 202 202
query7 4211 592 330 330
query8 330 234 229 229
query9 8839 4154 4125 4125
query10 461 358 312 312
query11 5805 2365 2188 2188
query12 191 135 132 132
query13 1316 617 459 459
query14 6193 5533 5223 5223
query14_1 4528 4549 4540 4540
query15 226 208 188 188
query16 1026 484 451 451
query17 1186 779 649 649
query18 2485 487 381 381
query19 231 212 179 179
query20 143 137 136 136
query21 219 147 123 123
query22 13777 13646 13424 13424
query23 17390 16549 16280 16280
query23_1 16326 16303 16328 16303
query24 7403 1810 1294 1294
query24_1 1324 1313 1284 1284
query25 580 480 424 424
query26 1321 316 177 177
query27 2690 560 349 349
query28 4453 2038 2022 2022
query29 1018 628 508 508
query30 316 238 212 212
query31 1150 1069 943 943
query32 86 75 74 74
query33 538 359 292 292
query34 1187 1188 669 669
query35 795 809 705 705
query36 1438 1418 1273 1273
query37 157 117 101 101
query38 3236 3222 3104 3104
query39 945 921 903 903
query39_1 898 881 889 881
query40 233 152 129 129
query41 65 61 63 61
query42 113 116 118 116
query43 345 353 314 314
query44
query45 216 209 202 202
query46 1095 1240 759 759
query47 2363 2370 2220 2220
query48 408 425 302 302
query49 652 495 404 404
query50 1095 356 274 274
query51 4359 4323 4261 4261
query52 108 111 98 98
query53 264 292 208 208
query54 309 298 276 276
query55 94 93 92 92
query56 314 311 324 311
query57 1441 1404 1323 1323
query58 306 290 298 290
query59 1619 1705 1448 1448
query60 338 334 315 315
query61 159 148 157 148
query62 715 645 582 582
query63 248 209 209 209
query64 2420 807 627 627
query65
query66 1748 479 365 365
query67 29737 29688 29526 29526
query68
query69 465 352 294 294
query70 1051 1028 1029 1028
query71 322 284 286 284
query72 2920 2680 2441 2441
query73 851 749 454 454
query74 5110 4986 4797 4797
query75 2711 2602 2265 2265
query76 2349 1222 820 820
query77 427 431 347 347
query78 12370 12467 11886 11886
query79 1500 1024 793 793
query80 689 551 470 470
query81 456 285 248 248
query82 1384 167 126 126
query83 363 287 260 260
query84 255 149 143 143
query85 886 549 468 468
query86 406 329 323 323
query87 3440 3383 3250 3250
query88 3713 2790 2798 2790
query89 451 396 353 353
query90 1873 200 190 190
query91 178 170 140 140
query92 82 76 76 76
query93 1450 1569 846 846
query94 566 366 323 323
query95 672 387 431 387
query96 1056 827 362 362
query97 2742 2740 2645 2645
query98 242 237 231 231
query99 1162 1179 1040 1040
Total cold run time: 254882 ms
Total hot run time: 171445 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](log) Add label and txn_id columns to audit_log for transactional SQL statements [doris]
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow commented on code in PR #63865:
URL: https://github.com/apache/doris/pull/63865#discussion_r3332230953
##########
fe/fe-core/src/main/java/org/apache/doris/catalog/InternalSchema.java:
##########
@@ -230,6 +230,11 @@ public class InternalSchema {
ScalarType.createType(PrimitiveType.STRING), ColumnNullableType.NULLABLE));
AUDIT_SCHEMA.add(new ColumnDef("compute_group",
ScalarType.createType(PrimitiveType.STRING), ColumnNullableType.NULLABLE));
+ // transaction info
+ AUDIT_SCHEMA.add(new ColumnDef("label",
+ ScalarType.createVarchar(Config.label_regex_length), ColumnNullableType.NULLABLE));
Review Comment:
the name 'label' is too general, maybe `load_label` is a better name?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](log) Add label and txn_id columns to audit_log for transactional SQL statements [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #63865:
URL: https://github.com/apache/doris/pull/63865#discussion_r3333467294
##########
fe/fe-core/src/main/java/org/apache/doris/load/DeleteHandler.java:
##########
@@ -144,6 +144,11 @@ public void process(Database targetDb, OlapTable targetTbl, List<Partition> sele
deleteJob.await();
String commitMsg = deleteJob.commit();
execState.setOk(0, 0, commitMsg);
+ ConnectContext ctx = ConnectContext.get();
+ if (ctx != null) {
+ ctx.setOrUpdateInsertResult(deleteJob.getTransactionId(), deleteJob.getLabel(),
+ targetDb.getFullName(), targetTbl.getName(), TransactionStatus.VISIBLE, 0, 0);
Review Comment:
`DeleteJob.commit()` can return successfully with the transaction still in `COMMITTED` when `commitAndPublishTransaction(...)` times out before publish finishes; it even puts `status:'COMMITTED'` in `commitMsg` for that case. This new insert result always records `TransactionStatus.VISIBLE`, so after a valid committed-but-not-visible DELETE, `SHOW LAST INSERT` (and any later consumer of `ctx.getInsertResult()`) reports VISIBLE while the transaction is not visible yet. Please derive the status from the commit result/transaction state instead of hardcoding VISIBLE.
##########
regression-test/suites/audit/test_audit_log_label_txn_id.groovy:
##########
@@ -0,0 +1,134 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_audit_log_label_txn_id","nonConcurrent") {
+ try {
+ sql "set global enable_audit_plugin = true"
+ } catch (Exception e) {
+ log.warn("skip this case, because " + e.getMessage())
+ assertTrue(e.getMessage().toUpperCase().contains("ADMIN"))
+ return
+ }
+
+ sql "drop table if exists audit_label_txn_test"
+ sql """
+ CREATE TABLE `audit_label_txn_test` (
+ `id` bigint,
+ `name` varchar(32)
+ ) ENGINE=OLAP
+ UNIQUE KEY(`id`)
+ COMMENT 'OLAP'
+ DISTRIBUTED BY HASH(`id`) BUCKETS 1
+ PROPERTIES (
+ "replication_allocation" = "tag.location.default: 1"
+ )
+ """
+
+ def auditSchema = sql """desc internal.__internal_schema.audit_log"""
+ def columnNames = auditSchema.collect { it[0].toString().toLowerCase() }
+ assertTrue(columnNames.contains("label"), "audit_log should contain label column")
+ assertTrue(columnNames.contains("txn_id"), "audit_log should contain txn_id column")
+
+ sql "truncate table __internal_schema.audit_log"
+
+ def uniqueLabel = "audit_test_" + UUID.randomUUID().toString().replace("-", "_")
+
+ // INSERT with label
+ sql "insert into audit_label_txn_test with label ${uniqueLabel} values (1, 'alice')"
+
+ // INSERT without explicit label (auto-generated)
+ sql "insert into audit_label_txn_test values (2, 'bob')"
+
+ // UPDATE
+ sql "update audit_label_txn_test set name = 'charlie' where id = 2"
+
+ // DELETE
+ sql "delete from audit_label_txn_test where id = 1"
+
+ // SELECT (should have empty label and -1 txn_id)
+ sql "select * from audit_label_txn_test"
+
+ Thread.sleep(6000)
+ sql """call flush_audit_log()"""
+
+ // Check INSERT with label has non-empty label and valid txn_id
+ def retry = 180
+ def insertWithLabelRes = sql "select label, txn_id from __internal_schema.audit_log where stmt_type = 'INSERT' and label = '${uniqueLabel}' order by time desc limit 1"
+ while (insertWithLabelRes.isEmpty()) {
+ if (retry-- < 0) {
+ throw new RuntimeException("It has retried a few but still failed to find INSERT with label")
+ }
+ sleep(3000)
+ insertWithLabelRes = sql "select label, txn_id from __internal_schema.audit_log where stmt_type = 'INSERT' and label = '${uniqueLabel}' order by time desc limit 1"
+ }
+ assertTrue(insertWithLabelRes[0][0].toString().length() > 0, "INSERT with label should have non-empty label")
+ assertTrue(Long.parseLong(insertWithLabelRes[0][1].toString()) > 0, "INSERT should have valid txn_id")
+
+ // Check INSERT without explicit label still has auto-generated label and valid txn_id
+ retry = 180
+ def insertNoLabelRes = sql "select label, txn_id from __internal_schema.audit_log where stmt_type = 'INSERT' and stmt like '%audit_label_txn_test%' and label != '${uniqueLabel}' order by time desc limit 1"
+ while (insertNoLabelRes.isEmpty()) {
+ if (retry-- < 0) {
+ throw new RuntimeException("It has retried a few but still failed to find INSERT without label")
+ }
+ sleep(3000)
+ insertNoLabelRes = sql "select label, txn_id from __internal_schema.audit_log where stmt_type = 'INSERT' and stmt like '%audit_label_txn_test%' and label != '${uniqueLabel}' order by time desc limit 1"
+ }
+ assertTrue(insertNoLabelRes[0][0].toString().length() > 0, "INSERT without explicit label should have auto-generated label")
+ assertTrue(Long.parseLong(insertNoLabelRes[0][1].toString()) > 0, "INSERT should have valid txn_id")
+
+ // Check UPDATE has label and txn_id
+ retry = 180
Review Comment:
This UPDATE check is not scoped to the statement/table generated by this test, and the DELETE check below has the same problem. `truncate __internal_schema.audit_log` only clears persisted rows; it does not clear events already buffered in the audit loader, so `call flush_audit_log()` can reinsert older UPDATE/DELETE audit events from before the truncate. In that case these queries can pass with an unrelated row and fail to verify that this PR populated label/txn_id for `audit_label_txn_test`. Please filter UPDATE/DELETE by this test's statement/table (or another unique marker) the same way the INSERT-without-label check does.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](log) Add label and txn_id columns to audit_log for transactional SQL statements [doris]
Posted by "heguanhui (via GitHub)" <gi...@apache.org>.
heguanhui commented on PR #63865:
URL: https://github.com/apache/doris/pull/63865#issuecomment-4595417069
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](log) Add label and txn_id columns to audit_log for transactional SQL statements [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63865:
URL: https://github.com/apache/doris/pull/63865#issuecomment-4566670927
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](log) Add label and txn_id columns to audit_log for transactional SQL statements [doris]
Posted by "heguanhui (via GitHub)" <gi...@apache.org>.
heguanhui commented on PR #63865:
URL: https://github.com/apache/doris/pull/63865#issuecomment-4566683483
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org