You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "heguanhui (via GitHub)" <gi...@apache.org> on 2026/05/28 17:34:53 UTC

[PR] [feature](log) Add label and txn_id columns to audit_log for transactional SQL statements [doris]

heguanhui opened a new pull request, #63865:
URL: https://github.com/apache/doris/pull/63865

   ## What problem does this PR solve?
   
   Issue Number: close #xxx
   
   Related PR: #xxx
   
   Problem Summary: The `audit_log` internal table did not record transaction information (`label` and `txn_id`) for transactional SQL statements (INSERT/UPDATE/DELETE/MERGE_INTO). Users and DBAs could not trace which transaction a particular DML operation belonged to, making it difficult to correlate audit log entries with transaction metadata. Additionally, the old-style DELETE path (via `DeleteHandler`) did not call `setOrUpdateInsertResult()`, causing `label`/`txn_id` to be missing even if they were recorded.
   
   ## What is changed and how it works?
   
   1. **Add `label` and `txn_id` fields to `AuditEvent`** — Two new `@AuditField` annotated public fields with corresponding builder setters.
   2. **Populate label/txn_id in `AuditLogHelper.logAuditLogImpl()`** — Only for transactional statement types (INSERT/UPDATE/DELETE/MERGE_INTO), read from `ConnectContext.insertResult`. Non-transactional statements (e.g., SELECT) get empty label and -1 txn_id.
   3. **Add `label` and `txn_id` columns to `InternalSchema.AUDIT_SCHEMA`** — `label` as `varchar(128)`, `txn_id` as `bigint`. Existing audit_log tables will be auto-altered by `InternalSchemaInitializer.alterAuditSchemaIfNeeded()`.
   4. **Serialize label/txn_id in `AuditLoader.fillLogBuffer()`** — Append the two new fields to the CSV buffer for stream load into audit_log.
   5. **Remove duplicate `label` field from `LoadAuditEvent` and `StreamLoadAuditEvent`** — These subclasses now inherit `label` from the base `AuditEvent`, avoiding `Class.getFields()` reflection returning duplicate fields.
   6. **Fix old-style DELETE path in `DeleteHandler.process()`** — Call `ctx.setOrUpdateInsertResult()` after successful commit so that label/txn_id are available for audit logging.
   
   ## Release note
   
   Add `label` (varchar(128)) and `txn_id` (bigint) columns to the `audit_log` internal table. These columns record the transaction label and transaction ID for INSERT, UPDATE, DELETE, and MERGE INTO statements, enabling better transaction tracing and audit analysis.
   
   ## Check List (For Author)
   
   - Test: Regression test / Unit Test
       - Regression test: `test_audit_log_label_txn_id.groovy` (INSERT with/without label, UPDATE, DELETE, SELECT)
       - Unit Test: `AuditLogHelperTest` (4 tests), `AuditLogBuilderTest` (8 tests), `AuditEventProcessorTest`, `InternalSchemaInitializerTest`, `InternalSchemaAlterTest`, `AuditLogWorkloadGroupTest` (20 tests total, all passed)
   - Behavior changed: Yes — audit_log table now has two additional columns; old-style DELETE now populates insertResult
   - Does this need documentation: Yes (will add separately)
   
   ## Check List (For Reviewer)
   
   - [ ] I have added test cases for this bug fix or new feature
   - [ ] This PR will not cause performance regression
   - [ ] This PR will not break existing features


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](log) Add label and txn_id columns to audit_log for transactional SQL statements [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63865:
URL: https://github.com/apache/doris/pull/63865#issuecomment-4595969403

   # FE UT Coverage Report
   Increment line coverage `  59.62% (31/52)` :tada:
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/63865_d62e750d2dcd7dca12aa56d2ca69e36c07e7b5c3/fe_increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/63865_d62e750d2dcd7dca12aa56d2ca69e36c07e7b5c3/fe_report/index.html)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](log) Add label and txn_id columns to audit_log for transactional SQL statements [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63865:
URL: https://github.com/apache/doris/pull/63865#issuecomment-4567093625

   
   <details>
   <summary>TPC-DS: <b>Total hot run time: 171053 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit 32708331fac5f1d28ec4781cc4e9a16a5f8e3014, data reload: false
   
   query5	4326	645	512	512
   query6	340	216	199	199
   query7	4224	524	313	313
   query8	367	243	227	227
   query9	8802	4001	3977	3977
   query10	448	334	300	300
   query11	5751	2402	2189	2189
   query12	177	127	120	120
   query13	1302	573	430	430
   query14	6096	5442	5133	5133
   query14_1	4427	4473	4438	4438
   query15	211	206	184	184
   query16	1005	454	418	418
   query17	1113	736	581	581
   query18	2430	479	360	360
   query19	217	208	169	169
   query20	134	128	128	128
   query21	215	144	115	115
   query22	13615	13569	13458	13458
   query23	17287	16625	16270	16270
   query23_1	16309	16362	16490	16362
   query24	7573	1760	1307	1307
   query24_1	1315	1299	1317	1299
   query25	592	513	443	443
   query26	1314	320	175	175
   query27	2731	588	349	349
   query28	4470	2021	2005	2005
   query29	990	673	520	520
   query30	307	247	203	203
   query31	1131	1071	976	976
   query32	91	76	75	75
   query33	559	355	306	306
   query34	1190	1116	660	660
   query35	775	794	699	699
   query36	1416	1427	1225	1225
   query37	157	107	92	92
   query38	3230	3181	3075	3075
   query39	919	911	902	902
   query39_1	904	886	862	862
   query40	236	147	127	127
   query41	71	73	68	68
   query42	110	110	122	110
   query43	326	334	284	284
   query44	
   query45	219	207	199	199
   query46	1074	1182	719	719
   query47	2385	2384	2264	2264
   query48	398	410	313	313
   query49	656	510	430	430
   query50	962	369	255	255
   query51	4336	4307	4395	4307
   query52	106	108	93	93
   query53	257	276	207	207
   query54	331	283	271	271
   query55	95	93	87	87
   query56	313	319	310	310
   query57	1427	1410	1334	1334
   query58	313	275	270	270
   query59	1599	1696	1425	1425
   query60	320	320	294	294
   query61	159	155	153	153
   query62	704	647	585	585
   query63	238	192	206	192
   query64	2388	809	649	649
   query65	
   query66	1707	489	350	350
   query67	29986	29725	29519	29519
   query68	
   query69	440	341	299	299
   query70	1018	989	1004	989
   query71	294	271	263	263
   query72	3062	2705	2385	2385
   query73	822	734	409	409
   query74	5146	4943	4770	4770
   query75	2698	2607	2292	2292
   query76	2302	1137	755	755
   query77	410	417	330	330
   query78	12471	12423	11922	11922
   query79	1441	1014	744	744
   query80	1378	543	451	451
   query81	489	280	238	238
   query82	1352	161	121	121
   query83	337	274	246	246
   query84	262	145	112	112
   query85	928	537	451	451
   query86	449	352	311	311
   query87	3439	3359	3192	3192
   query88	3589	2692	2694	2692
   query89	467	382	346	346
   query90	1849	178	180	178
   query91	179	178	163	163
   query92	75	78	75	75
   query93	1511	1467	892	892
   query94	690	365	274	274
   query95	688	462	339	339
   query96	1007	781	357	357
   query97	2732	2755	2645	2645
   query98	236	224	224	224
   query99	1222	1169	1028	1028
   Total cold run time: 255125 ms
   Total hot run time: 171053 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](log) Add label and txn_id columns to audit_log for transactional SQL statements [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #63865:
URL: https://github.com/apache/doris/pull/63865#discussion_r3332258586


##########
regression-test/suites/audit/test_audit_log_label_txn_id.groovy:
##########
@@ -0,0 +1,134 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_audit_log_label_txn_id","nonConcurrent") {
+    try {

Review Comment:
   This test enables the global audit plugin but only disables it at the very end. Any failure between this line and the final `set global enable_audit_plugin = false` leaves the global setting enabled for later suites in the same regression run, which can make unrelated tests non-deterministic and keep writing audit rows. Please restore the original/global value in a `finally` block (or otherwise guarantee cleanup) around the body after the privilege check succeeds.



##########
fe/fe-core/src/main/java/org/apache/doris/qe/AuditLogHelper.java:
##########
@@ -69,12 +69,18 @@ public class AuditLogHelper {
 
     private static final Logger LOG = LogManager.getLogger(AuditLogHelper.class);
     private static final Set<String> LOG_PLAN_INFO_TYPES = Sets.newTreeSet(String.CASE_INSENSITIVE_ORDER);
+    private static final Set<String> TXN_STMT_TYPES = Sets.newTreeSet(String.CASE_INSENSITIVE_ORDER);
 
     static {

Review Comment:
   `ConnectContext.getInsertResult()` is the session's last insert result used by `SHOW LAST INSERT`, and it is not cleared at statement start. That means an audited transactional statement can inherit stale metadata from a previous successful insert/update/delete in the same connection. For example: run a successful `insert` so `insertResult` is populated, then run an `insert` that fails during analysis/execution before `setOrUpdateInsertResult()` is called, or a `delete` that reaches `processEmptyRelation()` and reports txnId `-1` without updating the context. `stmtType` is still `INSERT`/`DELETE`, so this block writes the previous statement's label/txn_id into the current audit row. The audit fields need to be scoped to the current statement/result, or `insertResult` must be cleared/validated for the current statement before it is used here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](log) Add label and txn_id columns to audit_log for transactional SQL statements [doris]

Posted by "heguanhui (via GitHub)" <gi...@apache.org>.
heguanhui commented on code in PR #63865:
URL: https://github.com/apache/doris/pull/63865#discussion_r3332244141


##########
fe/fe-core/src/main/java/org/apache/doris/catalog/InternalSchema.java:
##########
@@ -230,6 +230,11 @@ public class InternalSchema {
                 ScalarType.createType(PrimitiveType.STRING), ColumnNullableType.NULLABLE));
         AUDIT_SCHEMA.add(new ColumnDef("compute_group",
                 ScalarType.createType(PrimitiveType.STRING), ColumnNullableType.NULLABLE));
+        // transaction info
+        AUDIT_SCHEMA.add(new ColumnDef("label",
+                ScalarType.createVarchar(Config.label_regex_length), ColumnNullableType.NULLABLE));

Review Comment:
   Your proposal is very good. I will refer to your suggestion and adjust the code accordingly



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](log) Add label and txn_id columns to audit_log for transactional SQL statements [doris]

Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow commented on PR #63865:
URL: https://github.com/apache/doris/pull/63865#issuecomment-4608971457

   /review


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](log) Add label and txn_id columns to audit_log for transactional SQL statements [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63865:
URL: https://github.com/apache/doris/pull/63865#issuecomment-4595615899

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 29155 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit d62e750d2dcd7dca12aa56d2ca69e36c07e7b5c3, data reload: false
   
   ------ Round 1 ----------------------------------
   orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
   ============================================
   q1	17620	4249	4212	4212
   q2	q3	10778	1418	841	841
   q4	4687	477	348	348
   q5	7572	888	580	580
   q6	192	183	141	141
   q7	835	839	650	650
   q8	9350	1603	1716	1603
   q9	6225	4512	4488	4488
   q10	6869	1839	1520	1520
   q11	451	280	259	259
   q12	646	436	303	303
   q13	18122	3393	2757	2757
   q14	272	258	243	243
   q15	q16	828	768	706	706
   q17	1016	989	996	989
   q18	6902	5691	5434	5434
   q19	1309	1322	1108	1108
   q20	512	415	261	261
   q21	5902	2671	2403	2403
   q22	441	369	309	309
   Total cold run time: 100529 ms
   Total hot run time: 29155 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
   ============================================
   q1	4559	4479	4503	4479
   q2	q3	4550	4987	4354	4354
   q4	2156	2261	1415	1415
   q5	4562	4443	4452	4443
   q6	250	185	135	135
   q7	1801	2242	1795	1795
   q8	2774	2357	2277	2277
   q9	8103	8056	8081	8056
   q10	4790	4780	4334	4334
   q11	612	465	409	409
   q12	767	894	639	639
   q13	3299	3687	2989	2989
   q14	306	332	303	303
   q15	q16	786	726	634	634
   q17	1392	1363	1345	1345
   q18	8299	7506	7254	7254
   q19	1190	1092	1121	1092
   q20	2226	2245	1967	1967
   q21	5408	4767	4750	4750
   q22	545	463	413	413
   Total cold run time: 58375 ms
   Total hot run time: 53083 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](log) Add label and txn_id columns to audit_log for transactional SQL statements [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63865:
URL: https://github.com/apache/doris/pull/63865#issuecomment-4609426521

   # FE Regression Coverage Report
   Increment line coverage `  3.98% (47/1181)` :tada:
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/63865_d62e750d2dcd7dca12aa56d2ca69e36c07e7b5c3_merge_fe/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/63865_d62e750d2dcd7dca12aa56d2ca69e36c07e7b5c3_merge_fe/report/index.html)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](log) Add label and txn_id columns to audit_log for transactional SQL statements [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63865:
URL: https://github.com/apache/doris/pull/63865#issuecomment-4568062105

   # FE Regression Coverage Report
   Increment line coverage `  96.15% (25/26)` :tada:
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/63865_32708331fac5f1d28ec4781cc4e9a16a5f8e3014_merge_fe/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/63865_32708331fac5f1d28ec4781cc4e9a16a5f8e3014_merge_fe/report/index.html)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](log) Add label and txn_id columns to audit_log for transactional SQL statements [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63865:
URL: https://github.com/apache/doris/pull/63865#issuecomment-4567012305

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 31029 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit 32708331fac5f1d28ec4781cc4e9a16a5f8e3014, data reload: false
   
   ------ Round 1 ----------------------------------
   orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
   ============================================
   q1	17820	3981	3920	3920
   q2	q3	10812	1380	803	803
   q4	4682	473	351	351
   q5	7586	2237	2071	2071
   q6	377	172	135	135
   q7	949	791	626	626
   q8	9541	1756	1563	1563
   q9	7130	4964	4941	4941
   q10	6443	2226	1836	1836
   q11	445	266	236	236
   q12	691	430	295	295
   q13	18238	3349	2786	2786
   q14	254	259	237	237
   q15	q16	819	776	705	705
   q17	996	912	851	851
   q18	6917	5933	5578	5578
   q19	1185	1195	1069	1069
   q20	523	396	258	258
   q21	5797	2563	2461	2461
   q22	421	354	307	307
   Total cold run time: 101626 ms
   Total hot run time: 31029 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
   ============================================
   q1	4331	4238	4259	4238
   q2	q3	4550	4940	4304	4304
   q4	2055	2202	1360	1360
   q5	4393	4292	4702	4292
   q6	257	203	142	142
   q7	2067	1798	1551	1551
   q8	2464	2100	2152	2100
   q9	8009	8003	7940	7940
   q10	4892	4713	4437	4437
   q11	575	425	370	370
   q12	771	749	528	528
   q13	3251	3613	2941	2941
   q14	302	303	276	276
   q15	q16	710	727	632	632
   q17	1343	1312	1334	1312
   q18	7990	7465	6748	6748
   q19	1129	1089	1082	1082
   q20	2219	2223	1939	1939
   q21	5200	4505	4395	4395
   q22	516	446	439	439
   Total cold run time: 57024 ms
   Total hot run time: 51026 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](log) Add label and txn_id columns to audit_log for transactional SQL statements [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63865:
URL: https://github.com/apache/doris/pull/63865#issuecomment-4596650066

   # FE Regression Coverage Report
   Increment line coverage `  90.38% (47/52)` :tada:
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/63865_d62e750d2dcd7dca12aa56d2ca69e36c07e7b5c3_merge_fe/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/63865_d62e750d2dcd7dca12aa56d2ca69e36c07e7b5c3_merge_fe/report/index.html)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](log) Add label and txn_id columns to audit_log for transactional SQL statements [doris]

Posted by "heguanhui (via GitHub)" <gi...@apache.org>.
heguanhui commented on PR #63865:
URL: https://github.com/apache/doris/pull/63865#issuecomment-4597848402

   /review


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](log) Add label and txn_id columns to audit_log for transactional SQL statements [doris]

Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow commented on PR #63865:
URL: https://github.com/apache/doris/pull/63865#issuecomment-4591577708

   /review


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](log) Add label and txn_id columns to audit_log for transactional SQL statements [doris]

Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow commented on PR #63865:
URL: https://github.com/apache/doris/pull/63865#issuecomment-4590319183

   /review


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](log) Add label and txn_id columns to audit_log for transactional SQL statements [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #63865:
URL: https://github.com/apache/doris/pull/63865#discussion_r3345877669


##########
fe/fe-core/src/main/java/org/apache/doris/qe/AuditLogHelper.java:
##########
@@ -279,6 +290,15 @@ private static void logAuditLogImpl(ConnectContext ctx, String origStmt, Stateme
                 .setCloudCluster(Strings.isNullOrEmpty(cluster) ? "UNKNOWN" : cluster)
                 .setWorkloadGroup(ctx.getWorkloadGroupName());
 
+        // load_label and txn_id for transactional statements (INSERT/UPDATE/DELETE)

Review Comment:
   This only reads `insertResult` from the local `ConnectContext`, but normal `INSERT`/`UPDATE`/`DELETE`/`MERGE_INTO` on a non-master FE are forwarded because these commands implement `ForwardWithSync`. The master-side `proxyExecute()` runs the DML and populates the proxy context's insert result, but `TMasterOpResult` only returns packet/status/affected rows/txnLoadInfo, and `FEOpExecutor.execute()` never copies label/txnId back into the original context. The observer/follower then audits the statement here with empty `load_label` and `txn_id = -1` even though the DML succeeded. Please propagate the insert result through the forward response, or otherwise set these audit fields from the forwarded execution result before logging.



##########
fe/fe-core/src/main/java/org/apache/doris/plugin/audit/LoadAuditEvent.java:
##########
@@ -65,8 +63,8 @@ public AuditEventBuilder setJobId(long jobId) {
             return this;
         }
 
-        public AuditEventBuilder setLabel(String label) {
-            auditEvent.label = label;
+        public AuditEventBuilder setLoadLabel(String loadLabel) {

Review Comment:
   `AuditLogBuilder.auditLoadLog()` still filters and emits `LOAD_ANNONATION_NAMES`, which contains `Label`, but this PR removed the only `@AuditField(value = "Label")` from `LoadAuditEvent` and stores the value in inherited `LoadLabel` instead. Because the builder map is keyed by annotation value, every `LOAD_SUCCEED` line in `fe.load.audit.log` will now contain `|Label=null` even when the load has a label. Please keep a `Label` audit field for load audit events or update the load audit builder/output name together with this rename.



##########
fe/fe-core/src/main/java/org/apache/doris/plugin/audit/StreamLoadAuditEvent.java:
##########
@@ -64,8 +62,8 @@ public AuditEventBuilder setEventType(EventType eventType) {
             return this;
         }
 
-        public AuditEventBuilder setLabel(String label) {
-            auditEvent.label = label;
+        public AuditEventBuilder setLoadLabel(String loadLabel) {

Review Comment:
   The stream-load text audit path has the same regression: `AuditLogBuilder.auditStreamLoadLog()` still looks for annotation `Label`, but this class no longer exposes any `@AuditField(value = "Label")`. As a result `fe.stream_load.audit.log` will print `|Label=null` for completed stream loads. Please keep the legacy `Label` audit field here or update the stream-load audit builder/output contract consistently.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](log) Add label and txn_id columns to audit_log for transactional SQL statements [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63865:
URL: https://github.com/apache/doris/pull/63865#issuecomment-4595698450

   
   <details>
   <summary>TPC-DS: <b>Total hot run time: 171445 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit d62e750d2dcd7dca12aa56d2ca69e36c07e7b5c3, data reload: false
   
   query5	4320	689	524	524
   query6	337	239	202	202
   query7	4211	592	330	330
   query8	330	234	229	229
   query9	8839	4154	4125	4125
   query10	461	358	312	312
   query11	5805	2365	2188	2188
   query12	191	135	132	132
   query13	1316	617	459	459
   query14	6193	5533	5223	5223
   query14_1	4528	4549	4540	4540
   query15	226	208	188	188
   query16	1026	484	451	451
   query17	1186	779	649	649
   query18	2485	487	381	381
   query19	231	212	179	179
   query20	143	137	136	136
   query21	219	147	123	123
   query22	13777	13646	13424	13424
   query23	17390	16549	16280	16280
   query23_1	16326	16303	16328	16303
   query24	7403	1810	1294	1294
   query24_1	1324	1313	1284	1284
   query25	580	480	424	424
   query26	1321	316	177	177
   query27	2690	560	349	349
   query28	4453	2038	2022	2022
   query29	1018	628	508	508
   query30	316	238	212	212
   query31	1150	1069	943	943
   query32	86	75	74	74
   query33	538	359	292	292
   query34	1187	1188	669	669
   query35	795	809	705	705
   query36	1438	1418	1273	1273
   query37	157	117	101	101
   query38	3236	3222	3104	3104
   query39	945	921	903	903
   query39_1	898	881	889	881
   query40	233	152	129	129
   query41	65	61	63	61
   query42	113	116	118	116
   query43	345	353	314	314
   query44	
   query45	216	209	202	202
   query46	1095	1240	759	759
   query47	2363	2370	2220	2220
   query48	408	425	302	302
   query49	652	495	404	404
   query50	1095	356	274	274
   query51	4359	4323	4261	4261
   query52	108	111	98	98
   query53	264	292	208	208
   query54	309	298	276	276
   query55	94	93	92	92
   query56	314	311	324	311
   query57	1441	1404	1323	1323
   query58	306	290	298	290
   query59	1619	1705	1448	1448
   query60	338	334	315	315
   query61	159	148	157	148
   query62	715	645	582	582
   query63	248	209	209	209
   query64	2420	807	627	627
   query65	
   query66	1748	479	365	365
   query67	29737	29688	29526	29526
   query68	
   query69	465	352	294	294
   query70	1051	1028	1029	1028
   query71	322	284	286	284
   query72	2920	2680	2441	2441
   query73	851	749	454	454
   query74	5110	4986	4797	4797
   query75	2711	2602	2265	2265
   query76	2349	1222	820	820
   query77	427	431	347	347
   query78	12370	12467	11886	11886
   query79	1500	1024	793	793
   query80	689	551	470	470
   query81	456	285	248	248
   query82	1384	167	126	126
   query83	363	287	260	260
   query84	255	149	143	143
   query85	886	549	468	468
   query86	406	329	323	323
   query87	3440	3383	3250	3250
   query88	3713	2790	2798	2790
   query89	451	396	353	353
   query90	1873	200	190	190
   query91	178	170	140	140
   query92	82	76	76	76
   query93	1450	1569	846	846
   query94	566	366	323	323
   query95	672	387	431	387
   query96	1056	827	362	362
   query97	2742	2740	2645	2645
   query98	242	237	231	231
   query99	1162	1179	1040	1040
   Total cold run time: 254882 ms
   Total hot run time: 171445 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](log) Add label and txn_id columns to audit_log for transactional SQL statements [doris]

Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow commented on code in PR #63865:
URL: https://github.com/apache/doris/pull/63865#discussion_r3332230953


##########
fe/fe-core/src/main/java/org/apache/doris/catalog/InternalSchema.java:
##########
@@ -230,6 +230,11 @@ public class InternalSchema {
                 ScalarType.createType(PrimitiveType.STRING), ColumnNullableType.NULLABLE));
         AUDIT_SCHEMA.add(new ColumnDef("compute_group",
                 ScalarType.createType(PrimitiveType.STRING), ColumnNullableType.NULLABLE));
+        // transaction info
+        AUDIT_SCHEMA.add(new ColumnDef("label",
+                ScalarType.createVarchar(Config.label_regex_length), ColumnNullableType.NULLABLE));

Review Comment:
   the name 'label' is too general, maybe `load_label` is a better name?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](log) Add label and txn_id columns to audit_log for transactional SQL statements [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #63865:
URL: https://github.com/apache/doris/pull/63865#discussion_r3333467294


##########
fe/fe-core/src/main/java/org/apache/doris/load/DeleteHandler.java:
##########
@@ -144,6 +144,11 @@ public void process(Database targetDb, OlapTable targetTbl, List<Partition> sele
             deleteJob.await();
             String commitMsg = deleteJob.commit();
             execState.setOk(0, 0, commitMsg);
+            ConnectContext ctx = ConnectContext.get();
+            if (ctx != null) {
+                ctx.setOrUpdateInsertResult(deleteJob.getTransactionId(), deleteJob.getLabel(),
+                        targetDb.getFullName(), targetTbl.getName(), TransactionStatus.VISIBLE, 0, 0);

Review Comment:
   `DeleteJob.commit()` can return successfully with the transaction still in `COMMITTED` when `commitAndPublishTransaction(...)` times out before publish finishes; it even puts `status:'COMMITTED'` in `commitMsg` for that case. This new insert result always records `TransactionStatus.VISIBLE`, so after a valid committed-but-not-visible DELETE, `SHOW LAST INSERT` (and any later consumer of `ctx.getInsertResult()`) reports VISIBLE while the transaction is not visible yet. Please derive the status from the commit result/transaction state instead of hardcoding VISIBLE.



##########
regression-test/suites/audit/test_audit_log_label_txn_id.groovy:
##########
@@ -0,0 +1,134 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_audit_log_label_txn_id","nonConcurrent") {
+    try {
+        sql "set global enable_audit_plugin = true"
+    } catch (Exception e) {
+        log.warn("skip this case, because " + e.getMessage())
+        assertTrue(e.getMessage().toUpperCase().contains("ADMIN"))
+        return
+    }
+
+    sql "drop table if exists audit_label_txn_test"
+    sql """
+        CREATE TABLE `audit_label_txn_test` (
+          `id` bigint,
+          `name` varchar(32)
+        ) ENGINE=OLAP
+        UNIQUE KEY(`id`)
+        COMMENT 'OLAP'
+        DISTRIBUTED BY HASH(`id`) BUCKETS 1
+        PROPERTIES (
+          "replication_allocation" = "tag.location.default: 1"
+        )
+        """
+
+    def auditSchema = sql """desc internal.__internal_schema.audit_log"""
+    def columnNames = auditSchema.collect { it[0].toString().toLowerCase() }
+    assertTrue(columnNames.contains("label"), "audit_log should contain label column")
+    assertTrue(columnNames.contains("txn_id"), "audit_log should contain txn_id column")
+
+    sql "truncate table __internal_schema.audit_log"
+
+    def uniqueLabel = "audit_test_" + UUID.randomUUID().toString().replace("-", "_")
+
+    // INSERT with label
+    sql "insert into audit_label_txn_test with label ${uniqueLabel} values (1, 'alice')"
+
+    // INSERT without explicit label (auto-generated)
+    sql "insert into audit_label_txn_test values (2, 'bob')"
+
+    // UPDATE
+    sql "update audit_label_txn_test set name = 'charlie' where id = 2"
+
+    // DELETE
+    sql "delete from audit_label_txn_test where id = 1"
+
+    // SELECT (should have empty label and -1 txn_id)
+    sql "select * from audit_label_txn_test"
+
+    Thread.sleep(6000)
+    sql """call flush_audit_log()"""
+
+    // Check INSERT with label has non-empty label and valid txn_id
+    def retry = 180
+    def insertWithLabelRes = sql "select label, txn_id from __internal_schema.audit_log where stmt_type = 'INSERT' and label = '${uniqueLabel}' order by time desc limit 1"
+    while (insertWithLabelRes.isEmpty()) {
+        if (retry-- < 0) {
+            throw new RuntimeException("It has retried a few but still failed to find INSERT with label")
+        }
+        sleep(3000)
+        insertWithLabelRes = sql "select label, txn_id from __internal_schema.audit_log where stmt_type = 'INSERT' and label = '${uniqueLabel}' order by time desc limit 1"
+    }
+    assertTrue(insertWithLabelRes[0][0].toString().length() > 0, "INSERT with label should have non-empty label")
+    assertTrue(Long.parseLong(insertWithLabelRes[0][1].toString()) > 0, "INSERT should have valid txn_id")
+
+    // Check INSERT without explicit label still has auto-generated label and valid txn_id
+    retry = 180
+    def insertNoLabelRes = sql "select label, txn_id from __internal_schema.audit_log where stmt_type = 'INSERT' and stmt like '%audit_label_txn_test%' and label != '${uniqueLabel}' order by time desc limit 1"
+    while (insertNoLabelRes.isEmpty()) {
+        if (retry-- < 0) {
+            throw new RuntimeException("It has retried a few but still failed to find INSERT without label")
+        }
+        sleep(3000)
+        insertNoLabelRes = sql "select label, txn_id from __internal_schema.audit_log where stmt_type = 'INSERT' and stmt like '%audit_label_txn_test%' and label != '${uniqueLabel}' order by time desc limit 1"
+    }
+    assertTrue(insertNoLabelRes[0][0].toString().length() > 0, "INSERT without explicit label should have auto-generated label")
+    assertTrue(Long.parseLong(insertNoLabelRes[0][1].toString()) > 0, "INSERT should have valid txn_id")
+
+    // Check UPDATE has label and txn_id
+    retry = 180

Review Comment:
   This UPDATE check is not scoped to the statement/table generated by this test, and the DELETE check below has the same problem. `truncate __internal_schema.audit_log` only clears persisted rows; it does not clear events already buffered in the audit loader, so `call flush_audit_log()` can reinsert older UPDATE/DELETE audit events from before the truncate. In that case these queries can pass with an unrelated row and fail to verify that this PR populated label/txn_id for `audit_label_txn_test`. Please filter UPDATE/DELETE by this test's statement/table (or another unique marker) the same way the INSERT-without-label check does.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](log) Add label and txn_id columns to audit_log for transactional SQL statements [doris]

Posted by "heguanhui (via GitHub)" <gi...@apache.org>.
heguanhui commented on PR #63865:
URL: https://github.com/apache/doris/pull/63865#issuecomment-4595417069

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](log) Add label and txn_id columns to audit_log for transactional SQL statements [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63865:
URL: https://github.com/apache/doris/pull/63865#issuecomment-4566670927

   
   Thank you for your contribution to Apache Doris.
   Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
   
   Please clearly describe your PR:
   1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
   2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
   3. What features were added. Why was this function added?
   4. Which code was refactored and why was this part of the code refactored?
   5. Which functions were optimized and what is the difference before and after the optimization?
       


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](log) Add label and txn_id columns to audit_log for transactional SQL statements [doris]

Posted by "heguanhui (via GitHub)" <gi...@apache.org>.
heguanhui commented on PR #63865:
URL: https://github.com/apache/doris/pull/63865#issuecomment-4566683483

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org