You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "qzsee (via GitHub)" <gi...@apache.org> on 2025/12/29 03:42:58 UTC
[PR] [Feature](privilege) support data mask in doris internal auth [doris]
qzsee opened a new pull request, #38301:
URL: https://github.com/apache/doris/pull/38301
## Proposed changes
Before, data masking could only be managed by ranger, and ranger was not applicable to users in most cases. doris built-in permission management did not have the management function of data masking, and now it is added.
supported data mask policy
```java
MASK_REDACT("Replace lowercase with 'x', uppercase with 'X', digits with '0'",
"regexp_replace(regexp_replace(regexp_replace({col},'([A-Z])', 'X'),'([a-z])','x'),'([0-9])','0')"),
MASK_SHOW_LAST_4("Show last 4 characters; replace rest with 'X'",
"LPAD(RIGHT({col}, 4), CHAR_LENGTH({col}), 'X')"),
MASK_SHOW_FIRST_4("Show first 4 characters; replace rest with 'x'",
"RPAD(LEFT({col}, 4), CHAR_LENGTH({col}), 'X')"),
MASK_HASH("Hash the value of a varchar with sha256",
"hex(sha2({col}, 256))"),
MASK_NULL("Replace with NULL", "NULL"),
MASK_DATE_SHOW_YEAR("Date: show only year",
"date_trunc({col}, 'year')"),
MASK_DEFAULT("Replace with data type default",
"");
```
```sql
create database test;
CREATE TABLE `stu` (
`id` INT NOT NULL,
`name` VARCHAR(200) NOT NULL,
`phone` INT NOT NULL,
`birth` DATETIME NOT NULL
) ENGINE=OLAP
DUPLICATE KEY(`id`, `name`)
DISTRIBUTED BY HASH(`id`) BUCKETS 1
PROPERTIES (
"replication_allocation" = "tag.location.default: 1"
);
```
1. add data mask policy
```sql
create data mask policy test on internal.test.stu.phone to 'jack' using MASK_DEFAULT;
create data mask policy test1 on internal.test.stu.id to 'jack' using MASK_NULL;
create data mask policy test2 on internal.test.stu.name to 'jack' using MASK_HASH;
```
2. show data mask policy
```sql
MySQL [test]> show data mask policy;
+------------+-------------+--------+-----------+------------+--------------+-----------------------+------+------+
| PolicyName | CatalogName | DbName | TableName | ColumnName | DataMaskType | DataMaskDef | User | Role |
+------------+-------------+--------+-----------+------------+--------------+-----------------------+------+------+
| test | internal | test | stu | phone | MASK_DEFAULT | | jack | NULL |
| test1 | internal | test | stu | id | MASK_NULL | NULL | jack | NULL |
| test2 | internal | test | stu | name | MASK_HASH | hex(sha2({col}, 256)) | jack | NULL |
+------------+-------------+--------+-----------+------------+--------------+-----------------------+------+------+
```
3. drop data mask policy
```sql
MySQL [test]> drop data mask policy test;
Query OK, 0 rows affected (0.01 sec)
MySQL [test]> show data mask policy;
+------------+-------------+--------+-----------+------------+--------------+-----------------------+------+------+
| PolicyName | CatalogName | DbName | TableName | ColumnName | DataMaskType | DataMaskDef | User | Role |
+------------+-------------+--------+-----------+------------+--------------+-----------------------+------+------+
| test1 | internal | test | stu | id | MASK_NULL | NULL | jack | NULL |
| test2 | internal | test | stu | name | MASK_HASH | hex(sha2({col}, 256)) | jack | NULL |
+------------+-------------+--------+-----------+------------+--------------+-----------------------+------+------+
```
4. explain
```sql
MySQL [test]> explain select * from stu;
+------------------------------------------------------------------------------+
| Explain String(Nereids Planner) |
+------------------------------------------------------------------------------+
| PLAN FRAGMENT 0 |
| OUTPUT EXPRS: |
| id[#4] |
| name[#5] |
| phone[#6] |
| birth[#7] |
| PARTITION: UNPARTITIONED |
| |
| HAS_COLO_PLAN_NODE: false |
| |
| VRESULT SINK |
| MYSQL_PROTOCAL |
| |
| 1:VEXCHANGE |
| offset: 0 |
| distribute expr lists: |
| |
| PLAN FRAGMENT 1 |
| |
| PARTITION: HASH_PARTITIONED: id[#0] |
| |
| HAS_COLO_PLAN_NODE: false |
| |
| STREAM DATA SINK |
| EXCHANGE ID: 01 |
| UNPARTITIONED |
| |
| 0:VOlapScanNode(68) |
| TABLE: test.stu(stu), PREAGGREGATION: ON |
| partitions=1/1 (stu) |
| tablets=1/1, tabletList=28797 |
| cardinality=1, avgRowSize=0.0, numNodes=1 |
| pushAggOp=NONE |
| final projections: NULL, hex(sha2(name[#1], 256)), phone[#2], birth[#3] |
| final project output tuple id: 1 |
+------------------------------------------------------------------------------+
```
Issue Number: close #xxx
<!--Describe your changes.-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature] support data mask in doris internal auth [doris]
Posted by "qzsee (via GitHub)" <gi...@apache.org>.
qzsee commented on PR #38301:
URL: https://github.com/apache/doris/pull/38301#issuecomment-2311394901
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature] support data mask in doris internal auth [doris]
Posted by "qzsee (via GitHub)" <gi...@apache.org>.
qzsee commented on PR #38301:
URL: https://github.com/apache/doris/pull/38301#issuecomment-2295504235
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature] support data mask in doris internal auth [doris]
Posted by "qzsee (via GitHub)" <gi...@apache.org>.
qzsee commented on code in PR #38301:
URL: https://github.com/apache/doris/pull/38301#discussion_r1699770058
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalCheckPolicy.java:
##########
@@ -230,4 +235,62 @@ public RelatedPolicy(Optional<Expression> rowPolicyFilter, Optional<List<NamedEx
this.dataMaskProjects = dataMaskProjects;
}
}
+
+ private Expression getMaskTypeDef(NereidsParser parser, DataMaskPolicy dataMaskPolicy, Slot slot) {
+ if (dataMaskPolicy instanceof DorisDataMaskPolicy) {
Review Comment:
Yes, it is better to put it in the data mask logic, but I did it because I don't want to change too much code. I'll put it in the data mask logic and change the code later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](privilege) support data mask in doris internal auth [doris]
Posted by "dqz123 (via GitHub)" <gi...@apache.org>.
dqz123 commented on PR #38301:
URL: https://github.com/apache/doris/pull/38301#issuecomment-2662531334
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](privilege) support data mask in doris internal auth [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38301:
URL: https://github.com/apache/doris/pull/38301#issuecomment-2315690140
<details>
<summary>TPC-DS: <b>Total hot run time: 193079 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 63e67853eeb83a5749b675fd264a663eb58b5fe6, data reload: false
query1 1276 894 870 870
query2 6427 2026 1943 1943
query3 10765 4113 4040 4040
query4 59877 24680 23394 23394
query5 5495 509 488 488
query6 408 165 168 165
query7 5906 301 301 301
query8 291 206 209 206
query9 9047 2493 2482 2482
query10 482 268 257 257
query11 18047 15050 15484 15050
query12 157 101 100 100
query13 1559 393 372 372
query14 10931 7403 6419 6419
query15 234 181 184 181
query16 7497 521 509 509
query17 1138 576 565 565
query18 1524 307 302 302
query19 275 154 143 143
query20 121 117 116 116
query21 209 103 108 103
query22 4549 4518 4559 4518
query23 34289 33624 33567 33567
query24 5900 2867 2839 2839
query25 561 411 391 391
query26 695 156 152 152
query27 1805 276 275 275
query28 3842 2108 2080 2080
query29 644 404 405 404
query30 238 150 155 150
query31 912 764 746 746
query32 81 55 56 55
query33 485 282 283 282
query34 853 483 470 470
query35 851 753 708 708
query36 1043 977 942 942
query37 150 97 89 89
query38 4065 3864 3940 3864
query39 1479 1389 1372 1372
query40 205 121 122 121
query41 48 47 48 47
query42 122 103 102 102
query43 518 473 475 473
query44 1086 745 754 745
query45 195 163 169 163
query46 1089 772 742 742
query47 1912 1796 1824 1796
query48 366 309 299 299
query49 771 450 443 443
query50 816 433 419 419
query51 7085 7106 7123 7106
query52 98 89 87 87
query53 253 181 178 178
query54 574 464 478 464
query55 81 78 78 78
query56 286 270 273 270
query57 1186 1056 1068 1056
query58 222 239 238 238
query59 3010 2952 2791 2791
query60 318 282 278 278
query61 126 120 125 120
query62 755 667 661 661
query63 213 185 191 185
query64 2979 772 786 772
query65 3226 3145 3148 3145
query66 685 344 353 344
query67 15585 15275 15266 15266
query68 3113 607 613 607
query69 441 284 288 284
query70 1136 1127 1035 1035
query71 374 276 282 276
query72 6479 4155 4273 4155
query73 759 325 331 325
query74 9117 8800 8946 8800
query75 3335 2676 2676 2676
query76 1374 1160 1022 1022
query77 535 326 323 323
query78 9764 9081 9044 9044
query79 1584 537 532 532
query80 844 512 504 504
query81 579 232 228 228
query82 263 150 145 145
query83 227 148 154 148
query84 269 76 73 73
query85 832 315 276 276
query86 362 291 268 268
query87 4249 4275 4204 4204
query88 3280 2337 2325 2325
query89 386 288 279 279
query90 2078 195 190 190
query91 123 101 98 98
query92 63 51 49 49
query93 1767 561 542 542
query94 852 301 303 301
query95 355 266 262 262
query96 599 270 268 268
query97 3215 3090 3071 3071
query98 219 207 206 206
query99 1816 1292 1264 1264
Total cold run time: 308307 ms
Total hot run time: 193079 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature] support data mask in doris internal auth [doris]
Posted by "qzsee (via GitHub)" <gi...@apache.org>.
qzsee commented on PR #38301:
URL: https://github.com/apache/doris/pull/38301#issuecomment-2303894124
run p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](privilege) support data mask in doris internal auth [doris]
Posted by "buu-nguyen (via GitHub)" <gi...@apache.org>.
buu-nguyen commented on PR #38301:
URL: https://github.com/apache/doris/pull/38301#issuecomment-2776910841
Looking forward to seeing this PR merged. In my opinion, this PR is the first step toward a complete data governance solution for Doris.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature] support data mask in doris internal auth [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38301:
URL: https://github.com/apache/doris/pull/38301#issuecomment-2295547565
<details>
<summary>TPC-DS: <b>Total hot run time: 197140 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2b0383da1930218dfb731890284147b74ced8996, data reload: false
query1 1331 911 891 891
query2 6642 2008 1969 1969
query3 10616 3848 3720 3720
query4 57471 23897 23467 23467
query5 5894 725 708 708
query6 548 211 230 211
query7 6387 338 348 338
query8 543 441 452 441
query9 9303 2591 2545 2545
query10 601 347 335 335
query11 17831 14970 15238 14970
query12 208 140 146 140
query13 1701 427 423 423
query14 11980 7456 7673 7456
query15 250 200 193 193
query16 7539 552 555 552
query17 1175 638 610 610
query18 2067 352 364 352
query19 293 172 169 169
query20 145 144 162 144
query21 251 142 142 142
query22 4546 4386 4276 4276
query23 34579 33956 33741 33741
query24 5854 3073 2982 2982
query25 574 441 436 436
query26 742 186 182 182
query27 1897 306 305 305
query28 3922 2221 2206 2206
query29 735 448 454 448
query30 223 202 189 189
query31 1032 870 865 865
query32 106 82 85 82
query33 537 352 353 352
query34 914 527 517 517
query35 890 805 805 805
query36 1126 989 972 972
query37 160 101 117 101
query38 3899 3870 4010 3870
query39 1564 1484 1442 1442
query40 232 153 156 153
query41 139 138 139 138
query42 143 114 115 114
query43 548 520 506 506
query44 1109 785 802 785
query45 229 197 196 196
query46 1133 778 776 776
query47 1915 1809 1818 1809
query48 414 333 331 331
query49 917 597 599 597
query50 872 464 484 464
query51 6886 6717 6804 6717
query52 122 109 110 109
query53 304 224 226 224
query54 612 512 502 502
query55 89 90 89 89
query56 324 309 324 309
query57 1218 1136 1139 1136
query58 311 313 301 301
query59 3112 2804 2948 2804
query60 363 330 322 322
query61 151 144 143 143
query62 791 707 707 707
query63 265 226 228 226
query64 3365 1845 1841 1841
query65 3251 3173 3154 3154
query66 965 672 717 672
query67 15178 14961 15008 14961
query68 8322 592 579 579
query69 707 459 349 349
query70 1182 1159 1157 1157
query71 562 324 320 320
query72 6765 2323 2060 2060
query73 845 359 352 352
query74 9092 8757 8841 8757
query75 3906 2730 2793 2730
query76 4595 1066 1053 1053
query77 851 505 449 449
query78 11120 9753 12778 9753
query79 8840 574 576 574
query80 1663 625 630 625
query81 620 262 259 259
query82 317 158 158 158
query83 358 212 213 212
query84 302 98 95 95
query85 816 355 367 355
query86 369 320 311 311
query87 4520 4117 4265 4117
query88 4390 2455 2452 2452
query89 443 327 323 323
query90 2304 237 235 235
query91 159 164 127 127
query92 86 80 77 77
query93 1077 556 548 548
query94 1101 331 329 329
query95 386 294 297 294
query96 593 292 290 290
query97 3227 3072 3097 3072
query98 247 233 227 227
query99 1569 1268 1305 1268
Total cold run time: 332368 ms
Total hot run time: 197140 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature] support data mask in doris internal auth [doris]
Posted by "qzsee (via GitHub)" <gi...@apache.org>.
qzsee commented on PR #38301:
URL: https://github.com/apache/doris/pull/38301#issuecomment-2303377475
run P0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature] support data mask in doris internal auth [doris]
Posted by "qzsee (via GitHub)" <gi...@apache.org>.
qzsee commented on PR #38301:
URL: https://github.com/apache/doris/pull/38301#issuecomment-2304019442
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature] support data mask in doris internal auth [doris]
Posted by "qzsee (via GitHub)" <gi...@apache.org>.
qzsee commented on PR #38301:
URL: https://github.com/apache/doris/pull/38301#issuecomment-2309175714
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature] support data mask in doris internal auth [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38301:
URL: https://github.com/apache/doris/pull/38301#issuecomment-2309224358
<details>
<summary>ClickBench: <b>Total hot run time: 30.04 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a489ed918b712b14923cea181ab90f7a8b885a9b, data reload: false
query1 0.04 0.05 0.04
query2 0.07 0.04 0.04
query3 0.22 0.05 0.05
query4 1.68 0.08 0.08
query5 0.50 0.49 0.51
query6 1.13 0.73 0.74
query7 0.02 0.02 0.02
query8 0.05 0.04 0.05
query9 0.54 0.48 0.48
query10 0.53 0.53 0.53
query11 0.16 0.12 0.12
query12 0.15 0.12 0.11
query13 0.63 0.59 0.60
query14 0.77 0.79 0.76
query15 0.84 0.81 0.82
query16 0.37 0.38 0.37
query17 1.04 1.08 1.05
query18 0.20 0.19 0.20
query19 1.88 1.83 1.83
query20 0.01 0.01 0.00
query21 15.40 0.68 0.68
query22 3.90 8.36 1.25
query23 18.27 1.36 1.20
query24 2.09 0.21 0.23
query25 0.16 0.08 0.08
query26 0.26 0.18 0.17
query27 0.08 0.07 0.07
query28 13.28 1.02 1.00
query29 12.63 3.28 3.27
query30 0.24 0.05 0.06
query31 2.87 0.41 0.39
query32 3.26 0.48 0.48
query33 2.96 2.98 2.99
query34 17.10 4.38 4.38
query35 4.46 4.44 4.43
query36 0.66 0.50 0.47
query37 0.18 0.15 0.16
query38 0.16 0.15 0.16
query39 0.04 0.04 0.04
query40 0.16 0.13 0.12
query41 0.10 0.05 0.05
query42 0.06 0.05 0.05
query43 0.05 0.04 0.05
Total cold run time: 109.2 s
Total hot run time: 30.04 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](privilege) support data mask in doris internal auth [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38301:
URL: https://github.com/apache/doris/pull/38301#issuecomment-2371234637
<details>
<summary>TPC-DS: <b>Total hot run time: 192868 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c38ecbfce784ec4c7da85e972b4973d7778e5c8c, data reload: false
query1 963 379 402 379
query2 6370 2128 2060 2060
query3 8688 193 204 193
query4 34050 23635 23521 23521
query5 3453 478 461 461
query6 268 164 165 164
query7 4186 304 311 304
query8 283 216 220 216
query9 9234 2691 2673 2673
query10 471 282 280 280
query11 18148 15288 15302 15288
query12 160 99 98 98
query13 1546 439 430 430
query14 9617 7165 7884 7165
query15 268 176 183 176
query16 8011 465 484 465
query17 1661 627 585 585
query18 2126 321 320 320
query19 388 163 165 163
query20 120 124 118 118
query21 223 113 117 113
query22 5036 4511 4552 4511
query23 35354 34667 34402 34402
query24 11181 2986 2981 2981
query25 623 418 422 418
query26 1193 167 164 164
query27 2219 298 300 298
query28 7555 2496 2447 2447
query29 830 468 440 440
query30 258 160 155 155
query31 1013 800 790 790
query32 102 54 56 54
query33 772 300 305 300
query34 914 517 507 507
query35 894 776 739 739
query36 1109 908 953 908
query37 150 96 93 93
query38 3964 3898 3890 3890
query39 1502 1422 1423 1422
query40 208 99 101 99
query41 52 51 50 50
query42 120 99 100 99
query43 550 492 506 492
query44 1319 808 815 808
query45 200 172 167 167
query46 1151 711 709 709
query47 1961 1846 1885 1846
query48 488 368 387 368
query49 893 411 423 411
query50 831 425 428 425
query51 7113 6987 6901 6901
query52 98 88 87 87
query53 285 186 192 186
query54 1242 471 470 470
query55 82 76 77 76
query56 281 262 273 262
query57 1236 1128 1121 1121
query58 249 242 224 224
query59 3358 3021 2973 2973
query60 284 268 270 268
query61 103 107 105 105
query62 839 685 676 676
query63 220 202 188 188
query64 4066 666 649 649
query65 3253 3204 3204 3204
query66 719 314 337 314
query67 15856 15618 15797 15618
query68 4923 573 562 562
query69 518 299 309 299
query70 1172 1137 1118 1118
query71 347 277 278 277
query72 7546 3954 4011 3954
query73 779 346 350 346
query74 10535 9009 8958 8958
query75 3473 2690 2690 2690
query76 3205 919 866 866
query77 625 300 290 290
query78 10063 9325 9319 9319
query79 1581 592 584 584
query80 2469 450 454 450
query81 569 245 246 245
query82 722 144 140 140
query83 301 135 137 135
query84 284 85 80 80
query85 1582 297 282 282
query86 421 270 275 270
query87 4368 4402 4517 4402
query88 3194 2487 2436 2436
query89 410 283 296 283
query90 1962 189 193 189
query91 184 144 145 144
query92 64 48 50 48
query93 1085 538 537 537
query94 1023 290 312 290
query95 357 271 262 262
query96 624 300 298 298
query97 3282 3163 3150 3150
query98 221 203 193 193
query99 1743 1284 1299 1284
Total cold run time: 301592 ms
Total hot run time: 192868 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](privilege) support data mask in doris internal auth [doris]
Posted by "dqz123 (via GitHub)" <gi...@apache.org>.
dqz123 commented on PR #38301:
URL: https://github.com/apache/doris/pull/38301#issuecomment-3696527433
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](privilege) support data mask in doris internal auth [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38301:
URL: https://github.com/apache/doris/pull/38301#issuecomment-2371204912
<details>
<summary>TPC-H: <b>Total hot run time: 40901 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c38ecbfce784ec4c7da85e972b4973d7778e5c8c, data reload: false
------ Round 1 ----------------------------------
q1 17938 7723 7300 7300
q2 2035 292 299 292
q3 12222 1070 1173 1070
q4 10485 772 717 717
q5 7782 2852 2790 2790
q6 241 153 148 148
q7 992 610 615 610
q8 9373 2020 2019 2019
q9 6587 6491 6428 6428
q10 6986 2303 2301 2301
q11 435 252 248 248
q12 412 218 219 218
q13 17787 2985 2972 2972
q14 232 212 212 212
q15 578 523 525 523
q16 672 616 630 616
q17 975 564 608 564
q18 7375 6507 6748 6507
q19 1396 1054 1008 1008
q20 601 291 281 281
q21 4114 3236 3109 3109
q22 1076 968 998 968
Total cold run time: 110294 ms
Total hot run time: 40901 ms
----- Round 2, with runtime_filter_mode=off -----
q1 7297 7286 7232 7232
q2 333 232 229 229
q3 3110 3000 2966 2966
q4 2090 1841 1831 1831
q5 5674 5689 5721 5689
q6 232 147 149 147
q7 2226 1811 1823 1811
q8 3383 3500 3538 3500
q9 8895 9051 8925 8925
q10 3505 3523 3583 3523
q11 602 501 499 499
q12 852 658 595 595
q13 8259 3175 3159 3159
q14 299 276 286 276
q15 577 538 533 533
q16 722 681 666 666
q17 1806 1642 1603 1603
q18 8329 7842 7925 7842
q19 1704 1598 1565 1565
q20 2137 1879 1920 1879
q21 5757 5391 5471 5391
q22 1155 1003 1045 1003
Total cold run time: 68944 ms
Total hot run time: 60864 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](privilege) support data mask in doris internal auth [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38301:
URL: https://github.com/apache/doris/pull/38301#issuecomment-2662639625
<details>
<summary>ClickBench: <b>Total hot run time: 30.4 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8f011cd6290513cf910b91ba1fd59477ca98c7db, data reload: false
query1 0.03 0.03 0.03
query2 0.08 0.03 0.04
query3 0.23 0.07 0.06
query4 1.61 0.10 0.11
query5 0.41 0.43 0.40
query6 1.16 0.66 0.66
query7 0.02 0.01 0.01
query8 0.04 0.03 0.03
query9 0.59 0.52 0.53
query10 0.58 0.57 0.58
query11 0.15 0.10 0.10
query12 0.14 0.10 0.11
query13 0.62 0.60 0.60
query14 2.68 2.72 2.71
query15 0.92 0.85 0.86
query16 0.38 0.38 0.37
query17 1.04 0.98 1.07
query18 0.22 0.19 0.19
query19 1.92 1.83 2.04
query20 0.01 0.02 0.01
query21 15.37 0.90 0.54
query22 0.75 1.22 0.76
query23 14.85 1.39 0.65
query24 11.55 0.89 0.58
query25 0.38 0.26 0.12
query26 0.90 0.19 0.15
query27 0.05 0.04 0.05
query28 5.98 0.76 0.43
query29 12.61 3.96 3.27
query30 0.25 0.08 0.06
query31 2.84 0.57 0.39
query32 3.22 0.54 0.47
query33 2.94 3.09 3.03
query34 15.70 5.10 4.51
query35 4.56 4.46 4.50
query36 0.66 0.49 0.48
query37 0.09 0.07 0.06
query38 0.06 0.04 0.03
query39 0.02 0.02 0.02
query40 0.17 0.13 0.13
query41 0.08 0.02 0.03
query42 0.03 0.02 0.02
query43 0.03 0.03 0.03
Total cold run time: 105.92 s
Total hot run time: 30.4 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature] support data mask in doris internal auth [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #38301:
URL: https://github.com/apache/doris/pull/38301#issuecomment-2301543317
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature] support data mask in doris internal auth [doris]
Posted by "qzsee (via GitHub)" <gi...@apache.org>.
qzsee commented on PR #38301:
URL: https://github.com/apache/doris/pull/38301#issuecomment-2303574424
> we will support `custom` type like ranger?
I'll add another pr later. I'm a little busy right now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature] support data mask in doris internal auth [doris]
Posted by "qzsee (via GitHub)" <gi...@apache.org>.
qzsee commented on PR #38301:
URL: https://github.com/apache/doris/pull/38301#issuecomment-2303594649
run p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature] support data mask in doris internal auth [doris]
Posted by "qzsee (via GitHub)" <gi...@apache.org>.
qzsee commented on PR #38301:
URL: https://github.com/apache/doris/pull/38301#issuecomment-2300337199
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature] support data mask in doris internal auth [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38301:
URL: https://github.com/apache/doris/pull/38301#issuecomment-2301290795
<details>
<summary>TPC-DS: <b>Total hot run time: 196962 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 97f81fcc30a6d38dfc9786c36ba7a755683492dd, data reload: false
query1 1337 905 872 872
query2 6647 2051 1974 1974
query3 10652 3858 3601 3601
query4 57036 25986 23517 23517
query5 6114 731 723 723
query6 550 219 216 216
query7 6424 347 330 330
query8 546 441 447 441
query9 9356 2522 2508 2508
query10 599 357 339 339
query11 18298 15305 15315 15305
query12 234 135 130 130
query13 1700 461 436 436
query14 11971 7353 7663 7353
query15 279 197 201 197
query16 7146 529 547 529
query17 1148 641 612 612
query18 1897 369 344 344
query19 327 171 172 171
query20 152 137 133 133
query21 250 143 142 142
query22 4404 4432 4425 4425
query23 34637 33569 34045 33569
query24 5895 2974 3002 2974
query25 590 447 440 440
query26 723 193 191 191
query27 1715 309 312 309
query28 3757 2191 2155 2155
query29 722 452 458 452
query30 231 194 196 194
query31 1051 876 867 867
query32 104 77 84 77
query33 524 353 366 353
query34 933 508 546 508
query35 861 791 783 783
query36 1116 948 971 948
query37 158 107 106 106
query38 4004 3890 3828 3828
query39 1497 1492 1464 1464
query40 241 157 154 154
query41 140 137 140 137
query42 134 119 115 115
query43 552 503 518 503
query44 1135 781 802 781
query45 221 196 195 195
query46 1121 793 757 757
query47 1928 1852 1853 1852
query48 404 349 330 330
query49 915 589 584 584
query50 870 476 470 470
query51 7255 7143 7133 7133
query52 121 109 113 109
query53 298 231 231 231
query54 629 519 519 519
query55 91 111 88 88
query56 344 317 319 317
query57 1202 1146 1133 1133
query58 308 310 294 294
query59 3084 2840 2885 2840
query60 360 328 323 323
query61 151 148 147 147
query62 778 701 702 701
query63 260 228 228 228
query64 3340 1859 1876 1859
query65 3269 3201 3218 3201
query66 1060 708 666 666
query67 15396 15160 14978 14978
query68 6282 602 595 595
query69 590 367 326 326
query70 1287 1143 1161 1143
query71 521 326 320 320
query72 6797 2307 2075 2075
query73 819 361 364 361
query74 9343 8888 8751 8751
query75 3657 2728 2713 2713
query76 3319 1118 988 988
query77 766 456 449 449
query78 10015 9094 9064 9064
query79 3890 556 559 556
query80 1512 600 615 600
query81 585 266 260 260
query82 848 166 156 156
query83 372 217 216 216
query84 295 102 102 102
query85 978 359 355 355
query86 454 276 320 276
query87 4483 4219 4222 4219
query88 4706 2510 2516 2510
query89 453 322 354 322
query90 2063 230 225 225
query91 152 126 125 125
query92 93 76 78 76
query93 4906 554 552 552
query94 828 309 325 309
query95 379 293 301 293
query96 638 296 290 290
query97 3243 3054 3072 3054
query98 241 227 227 227
query99 1592 1293 1304 1293
Total cold run time: 327299 ms
Total hot run time: 196962 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature] support data mask in doris internal auth [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38301:
URL: https://github.com/apache/doris/pull/38301#issuecomment-2293720715
<details>
<summary>TPC-H: <b>Total hot run time: 38388 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9ae811e9e64ebce1ec2fcfedaa276ad160c1162f, data reload: false
------ Round 1 ----------------------------------
q1 17867 4345 4317 4317
q2 2054 227 234 227
q3 11638 952 1092 952
q4 10524 716 721 716
q5 7771 2853 2813 2813
q6 272 165 161 161
q7 1032 674 656 656
q8 9590 2110 2116 2110
q9 8642 6547 6505 6505
q10 7094 2228 2209 2209
q11 500 276 275 275
q12 436 263 258 258
q13 17772 3010 2999 2999
q14 300 261 255 255
q15 576 530 542 530
q16 530 419 418 418
q17 996 734 709 709
q18 7511 6757 6753 6753
q19 6652 1016 1061 1016
q20 729 357 360 357
q21 4311 3205 3108 3108
q22 1130 1059 1044 1044
Total cold run time: 117927 ms
Total hot run time: 38388 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4572 4319 4313 4313
q2 438 312 317 312
q3 2884 2654 2753 2654
q4 1997 1727 1729 1727
q5 5666 5717 5634 5634
q6 259 150 154 150
q7 2134 1762 1757 1757
q8 3262 3469 3477 3469
q9 8762 8815 8735 8735
q10 3592 3352 3292 3292
q11 633 542 521 521
q12 855 682 630 630
q13 16953 3255 3193 3193
q14 321 295 290 290
q15 556 512 516 512
q16 519 452 441 441
q17 1849 1549 1534 1534
q18 8398 7803 7759 7759
q19 9714 1744 1514 1514
q20 2132 1918 1894 1894
q21 13599 5481 5494 5481
q22 1207 1080 1093 1080
Total cold run time: 90302 ms
Total hot run time: 56892 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](privilege) support data mask in doris internal auth [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #38301:
URL: https://github.com/apache/doris/pull/38301#issuecomment-3358576276
We're closing this PR because it hasn't been updated in a while.
This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and feel free a maintainer to remove the Stale tag!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature] support data mask in doris internal auth [doris]
Posted by "zddr (via GitHub)" <gi...@apache.org>.
zddr commented on code in PR #38301:
URL: https://github.com/apache/doris/pull/38301#discussion_r1696388612
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalCheckPolicy.java:
##########
@@ -230,4 +235,62 @@ public RelatedPolicy(Optional<Expression> rowPolicyFilter, Optional<List<NamedEx
this.dataMaskProjects = dataMaskProjects;
}
}
+
+ private Expression getMaskTypeDef(NereidsParser parser, DataMaskPolicy dataMaskPolicy, Slot slot) {
+ if (dataMaskPolicy instanceof DorisDataMaskPolicy) {
Review Comment:
I don't understand why it's necessary to determine if it's an internal Doris DataMaskPolicy? What is the purpose of this code? If necessary, it should also be placed inside the Doris data mask logic, and there should be no concern about what type of data mask it is
##########
fe/fe-core/src/main/java/org/apache/doris/policy/PolicyMgr.java:
##########
@@ -379,6 +389,40 @@ public RowPolicy getMatchTablePolicy(String ctlName, String dbName, String table
return mergeRowPolicies(res);
}
+ public DorisDataMaskPolicy getDataMaskPolicy(String ctlName, String dbName, String tableName, String colName, UserIdentity user) {
+ if (!hasDataMaskPolicy(ctlName, dbName, tableName, colName)) {
+ return null;
+ }
+ readLock();
+ try {
+ if (!hasDataMaskPolicy(ctlName, dbName, tableName, colName)) {
+ return null;
+ }
+ List<DorisDataMaskPolicy> dorisDataMaskPolicies = dataMaskPolicies.get(ctlName).get(dbName).get(tableName)
+ .get(colName);
+ Set<String> roles = Env.getCurrentEnv().getAccessManager().getAuth().getRolesByUserWithLdap(user).stream()
+ .map(role -> ClusterNamespace.getNameFromFullName(role.getRoleName())).collect(Collectors.toSet());
+
+ for (DorisDataMaskPolicy dataMaskPolicy : dorisDataMaskPolicies) {
+ if ((dataMaskPolicy.getUser() != null && dataMaskPolicy.getUser().getQualifiedUser()
Review Comment:
Unlike row policy, datamask can only return one piece of data.
So is it better to filter out users first and then find roles? This way, there can be a priority.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature] support data mask in doris internal auth [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38301:
URL: https://github.com/apache/doris/pull/38301#issuecomment-2304112964
<details>
<summary>ClickBench: <b>Total hot run time: 31.53 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 738658c3a73a9ce4dee98920146dc3dfe67cd382, data reload: false
query1 0.05 0.04 0.04
query2 0.08 0.04 0.04
query3 0.22 0.06 0.06
query4 1.67 0.07 0.08
query5 0.50 0.50 0.50
query6 1.13 0.73 0.73
query7 0.02 0.02 0.01
query8 0.05 0.04 0.04
query9 0.56 0.47 0.47
query10 0.53 0.54 0.54
query11 0.16 0.11 0.11
query12 0.15 0.12 0.12
query13 0.63 0.59 0.59
query14 0.78 0.80 0.80
query15 0.84 0.82 0.81
query16 0.37 0.38 0.35
query17 1.05 1.01 1.06
query18 0.21 0.20 0.21
query19 1.82 1.74 1.80
query20 0.01 0.00 0.01
query21 15.41 0.68 0.66
query22 4.21 6.61 2.52
query23 18.27 1.37 1.37
query24 2.09 0.22 0.25
query25 0.14 0.08 0.08
query26 0.27 0.18 0.17
query27 0.09 0.08 0.08
query28 13.22 1.03 1.02
query29 12.61 3.37 3.37
query30 0.25 0.05 0.07
query31 2.86 0.40 0.39
query32 3.25 0.48 0.48
query33 2.99 2.99 3.01
query34 16.91 4.40 4.38
query35 4.42 4.47 4.43
query36 0.66 0.49 0.48
query37 0.19 0.17 0.16
query38 0.16 0.16 0.15
query39 0.04 0.04 0.04
query40 0.16 0.13 0.13
query41 0.09 0.04 0.04
query42 0.06 0.05 0.05
query43 0.05 0.04 0.04
Total cold run time: 109.23 s
Total hot run time: 31.53 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature] support data mask in doris internal auth [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38301:
URL: https://github.com/apache/doris/pull/38301#issuecomment-2304078440
<details>
<summary>TPC-H: <b>Total hot run time: 37752 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 738658c3a73a9ce4dee98920146dc3dfe67cd382, data reload: false
------ Round 1 ----------------------------------
q1 17427 4331 4243 4243
q2 2032 183 175 175
q3 11236 1020 1103 1020
q4 10397 760 724 724
q5 7755 2785 2799 2785
q6 222 137 135 135
q7 987 599 598 598
q8 9327 2052 2060 2052
q9 7097 6519 6564 6519
q10 7005 2246 2250 2246
q11 472 244 249 244
q12 393 216 224 216
q13 18945 3079 3009 3009
q14 280 227 240 227
q15 520 493 486 486
q16 493 399 384 384
q17 969 651 689 651
q18 7335 6759 6813 6759
q19 1389 1020 990 990
q20 670 334 319 319
q21 3962 2972 2965 2965
q22 1154 1005 1017 1005
Total cold run time: 110067 ms
Total hot run time: 37752 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4391 4301 4305 4301
q2 377 278 257 257
q3 2875 2677 2678 2677
q4 1920 1607 1680 1607
q5 5532 5696 5685 5685
q6 227 136 140 136
q7 2311 1859 1828 1828
q8 3281 3384 3404 3384
q9 8834 8830 8818 8818
q10 3571 3364 3374 3364
q11 623 525 517 517
q12 812 665 688 665
q13 13716 3191 3157 3157
q14 317 312 301 301
q15 526 508 508 508
q16 486 466 447 447
q17 1822 1528 1518 1518
q18 8095 7709 7717 7709
q19 1723 1556 1568 1556
q20 2175 1916 1924 1916
q21 5773 5519 5618 5519
q22 1205 1118 1046 1046
Total cold run time: 70592 ms
Total hot run time: 56916 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature] support data mask in doris internal auth [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #38301:
URL: https://github.com/apache/doris/pull/38301#issuecomment-2304436987
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature] support data mask in doris internal auth [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38301:
URL: https://github.com/apache/doris/pull/38301#issuecomment-2295536993
<details>
<summary>TPC-H: <b>Total hot run time: 37996 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2b0383da1930218dfb731890284147b74ced8996, data reload: false
------ Round 1 ----------------------------------
q1 17865 4364 4292 4292
q2 2062 204 211 204
q3 11725 965 1196 965
q4 10505 785 709 709
q5 7781 2841 2844 2841
q6 264 161 157 157
q7 1033 651 673 651
q8 9599 2104 2120 2104
q9 8673 6576 6550 6550
q10 7084 2252 2249 2249
q11 518 272 274 272
q12 431 260 261 260
q13 17791 2987 2968 2968
q14 310 258 259 258
q15 560 536 536 536
q16 531 414 418 414
q17 987 669 666 666
q18 7360 6888 6736 6736
q19 5612 1067 1025 1025
q20 727 337 349 337
q21 3878 2775 2787 2775
q22 1106 1027 1039 1027
Total cold run time: 116402 ms
Total hot run time: 37996 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4698 4299 4287 4287
q2 404 318 294 294
q3 2854 2639 2611 2611
q4 1967 1702 1687 1687
q5 5650 5633 5748 5633
q6 233 145 144 144
q7 2188 1742 1750 1742
q8 3273 3512 3513 3512
q9 8778 8747 8714 8714
q10 3541 3382 3335 3335
q11 628 540 521 521
q12 864 664 679 664
q13 17294 3132 3179 3132
q14 318 300 286 286
q15 557 527 513 513
q16 513 442 460 442
q17 1868 1572 1525 1525
q18 8193 7984 7765 7765
q19 8899 1512 1586 1512
q20 2147 1902 1882 1882
q21 12157 5290 5314 5290
q22 1161 1070 1058 1058
Total cold run time: 88185 ms
Total hot run time: 56549 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature] support data mask in doris internal auth [doris]
Posted by "qzsee (via GitHub)" <gi...@apache.org>.
qzsee commented on PR #38301:
URL: https://github.com/apache/doris/pull/38301#issuecomment-2301209511
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature] support data mask in doris internal auth [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38301:
URL: https://github.com/apache/doris/pull/38301#issuecomment-2311441204
<details>
<summary>TPC-DS: <b>Total hot run time: 191767 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a489ed918b712b14923cea181ab90f7a8b885a9b, data reload: false
query1 1272 889 864 864
query2 6290 1935 1932 1932
query3 10581 4110 4030 4030
query4 59758 23886 23073 23073
query5 5305 528 538 528
query6 407 163 166 163
query7 5774 284 297 284
query8 286 214 208 208
query9 8842 2459 2446 2446
query10 480 273 252 252
query11 17949 15228 15214 15214
query12 159 111 120 111
query13 1571 387 368 368
query14 11155 7445 7376 7376
query15 242 172 171 171
query16 7490 459 483 459
query17 1133 560 559 559
query18 2096 290 299 290
query19 292 146 140 140
query20 120 112 115 112
query21 205 108 104 104
query22 4579 4424 4400 4400
query23 34386 33956 33507 33507
query24 5964 2842 2758 2758
query25 544 400 399 399
query26 691 161 160 160
query27 1773 301 271 271
query28 3894 2024 2009 2009
query29 676 405 398 398
query30 240 156 158 156
query31 921 773 720 720
query32 86 51 53 51
query33 458 279 281 279
query34 873 461 463 461
query35 835 718 733 718
query36 1051 937 958 937
query37 145 87 89 87
query38 3932 3934 3836 3836
query39 1448 1407 1408 1407
query40 202 116 117 116
query41 49 47 44 44
query42 113 95 95 95
query43 503 468 463 463
query44 1114 738 731 731
query45 197 165 164 164
query46 1081 732 735 732
query47 1868 1781 1788 1781
query48 380 299 296 296
query49 765 429 445 429
query50 821 416 426 416
query51 7296 7082 7150 7082
query52 105 95 93 93
query53 267 191 185 185
query54 610 482 474 474
query55 81 124 79 79
query56 287 268 264 264
query57 1189 1090 1102 1090
query58 219 227 232 227
query59 2872 2744 2663 2663
query60 290 280 268 268
query61 104 103 134 103
query62 741 640 627 627
query63 219 181 186 181
query64 3630 2096 1759 1759
query65 3209 3150 3117 3117
query66 676 322 323 322
query67 15324 15225 15159 15159
query68 2959 553 572 553
query69 398 274 286 274
query70 1202 1057 1093 1057
query71 360 283 278 278
query72 2547 2091 2043 2043
query73 696 320 319 319
query74 9216 8838 8770 8770
query75 3391 2695 2690 2690
query76 1439 948 957 948
query77 520 335 351 335
query78 9634 9127 9068 9068
query79 1029 532 530 530
query80 717 514 560 514
query81 453 230 224 224
query82 289 135 133 133
query83 174 149 150 149
query84 247 74 74 74
query85 672 288 282 282
query86 304 294 290 290
query87 4421 4204 4233 4204
query88 2949 2283 2296 2283
query89 385 285 279 279
query90 1918 199 198 198
query91 120 96 99 96
query92 63 53 50 50
query93 1056 518 527 518
query94 704 299 276 276
query95 329 263 262 262
query96 594 263 271 263
query97 3175 3125 3139 3125
query98 221 208 218 208
query99 1510 1276 1304 1276
Total cold run time: 301807 ms
Total hot run time: 191767 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature] support data mask in doris internal auth [doris]
Posted by "qzsee (via GitHub)" <gi...@apache.org>.
qzsee commented on code in PR #38301:
URL: https://github.com/apache/doris/pull/38301#discussion_r1699751511
##########
fe/fe-core/src/main/java/org/apache/doris/policy/PolicyMgr.java:
##########
@@ -379,6 +389,40 @@ public RowPolicy getMatchTablePolicy(String ctlName, String dbName, String table
return mergeRowPolicies(res);
}
+ public DorisDataMaskPolicy getDataMaskPolicy(String ctlName, String dbName, String tableName, String colName, UserIdentity user) {
+ if (!hasDataMaskPolicy(ctlName, dbName, tableName, colName)) {
+ return null;
+ }
+ readLock();
+ try {
+ if (!hasDataMaskPolicy(ctlName, dbName, tableName, colName)) {
+ return null;
+ }
+ List<DorisDataMaskPolicy> dorisDataMaskPolicies = dataMaskPolicies.get(ctlName).get(dbName).get(tableName)
+ .get(colName);
+ Set<String> roles = Env.getCurrentEnv().getAccessManager().getAuth().getRolesByUserWithLdap(user).stream()
+ .map(role -> ClusterNamespace.getNameFromFullName(role.getRoleName())).collect(Collectors.toSet());
+
+ for (DorisDataMaskPolicy dataMaskPolicy : dorisDataMaskPolicies) {
+ if ((dataMaskPolicy.getUser() != null && dataMaskPolicy.getUser().getQualifiedUser()
Review Comment:
Yes, there is a priority issue, which is not considered here, but only one piece of data is returned.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](privilege) support data mask in doris internal auth [doris]
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow commented on code in PR #38301:
URL: https://github.com/apache/doris/pull/38301#discussion_r1734308271
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/CreatePolicyCommand.java:
##########
@@ -80,6 +88,26 @@ public <R, C> R accept(PlanVisitor<R, C> visitor, C context) {
@Override
public void run(ConnectContext ctx, StmtExecutor executor) throws Exception {
+ if (policyType == PolicyTypeEnum.DATA_MASK) {
+ if (user != null) {
+ user.analyze();
+ if (user.isRootUser() || user.isAdminUser()) {
+ ErrorReport.reportAnalysisException(ErrorCode.ERR_TABLEACCESS_DENIED_ERROR, "CreatePolicyStmt",
+ user.getQualifiedUser(), user.getHost(), nameParts.get(0));
+ }
+ }
+ // check auth
+ if (!Env.getCurrentEnv().getAccessManager()
+ .checkGlobalPriv(ConnectContext.get(), PrivPredicate.GRANT)) {
+ ErrorReport.reportAnalysisException(ErrorCode.ERR_SPECIFIC_ACCESS_DENIED_ERROR,
+ PrivPredicate.GRANT.getPrivs().toString());
+ }
+ Policy policy = new DorisDataMaskPolicy(Env.getCurrentEnv().getNextId(), policyName, user, roleName,
+ nameParts.get(0), nameParts.get(1), nameParts.get(2), nameParts.get(3), dataMaskType);
+ Env.getCurrentEnv().getPolicyMgr().createPolicy(policy, ifNotExists);
+ return;
+ }
+ ctx.getSessionVariable().enableFallbackToOriginalPlannerOnce();
Review Comment:
no need to enableFallbackToOriginalPlannerOnce, MustFallbackException will ignore enable_fallback variable and use legacy planner to execute it directly
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/DropPolicyCommand.java:
##########
@@ -0,0 +1,78 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.plans.commands;
+
+import org.apache.doris.analysis.StmtType;
+import org.apache.doris.catalog.Env;
+import org.apache.doris.common.ErrorCode;
+import org.apache.doris.common.ErrorReport;
+import org.apache.doris.mysql.privilege.PrivPredicate;
+import org.apache.doris.nereids.exceptions.MustFallbackException;
+import org.apache.doris.nereids.trees.plans.PlanType;
+import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
+import org.apache.doris.policy.DropPolicyLog;
+import org.apache.doris.policy.PolicyTypeEnum;
+import org.apache.doris.qe.ConnectContext;
+import org.apache.doris.qe.StmtExecutor;
+
+/**
+ * drop policy command
+ */
+public class DropPolicyCommand extends Command implements ForwardWithSync {
+
+ private final PolicyTypeEnum policyType;
+ private final String policyName;
+ private final boolean ifExists;
+
+ /**
+ * constructor for drop command
+ */
+ public DropPolicyCommand(PolicyTypeEnum policyType, String policyName, boolean ifExists) {
+ super(PlanType.DROP_POLICY_COMMAND);
+ this.policyType = policyType;
+ this.policyName = policyName;
+ this.ifExists = ifExists;
+ }
+
+ @Override
+ public <R, C> R accept(PlanVisitor<R, C> visitor, C context) {
+ return visitor.visitDropPolicyCommand(this, context);
+ }
+
+ @Override
+ public void run(ConnectContext ctx, StmtExecutor executor) throws Exception {
+ if (policyType == PolicyTypeEnum.DATA_MASK) {
+ // check auth
+ if (!Env.getCurrentEnv().getAccessManager()
+ .checkGlobalPriv(ConnectContext.get(), PrivPredicate.GRANT)) {
+ ErrorReport.reportAnalysisException(ErrorCode.ERR_SPECIFIC_ACCESS_DENIED_ERROR,
+ PrivPredicate.GRANT.getPrivs().toString());
+ }
+ DropPolicyLog dropPolicyLog = new DropPolicyLog(policyType, policyName);
+ Env.getCurrentEnv().getPolicyMgr().dropPolicy(dropPolicyLog, ifExists);
+ return;
+ }
+ ctx.getSessionVariable().enableFallbackToOriginalPlannerOnce();
Review Comment:
ditto
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ShowPolicyCommand.java:
##########
@@ -0,0 +1,73 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.plans.commands;
+
+import org.apache.doris.analysis.UserIdentity;
+import org.apache.doris.catalog.Env;
+import org.apache.doris.common.ErrorCode;
+import org.apache.doris.common.ErrorReport;
+import org.apache.doris.mysql.privilege.PrivPredicate;
+import org.apache.doris.nereids.exceptions.MustFallbackException;
+import org.apache.doris.nereids.trees.plans.PlanType;
+import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
+import org.apache.doris.policy.PolicyTypeEnum;
+import org.apache.doris.qe.ConnectContext;
+import org.apache.doris.qe.ShowResultSet;
+import org.apache.doris.qe.StmtExecutor;
+
+/**
+ * show policy
+ */
+public class ShowPolicyCommand extends Command implements NoForward {
+
+ private final PolicyTypeEnum policyType;
+
+ private final UserIdentity user;
+
+ private final String roleName;
+
+ public ShowPolicyCommand(PolicyTypeEnum policyType, UserIdentity user, String roleName) {
+ super(PlanType.SHOW_POLICY_COMMAND);
+ this.policyType = policyType;
+ this.user = user;
+ this.roleName = roleName;
+ }
+
+ @Override
+ public <R, C> R accept(PlanVisitor<R, C> visitor, C context) {
+ return visitor.visitShowPolicyCommand(this, context);
+ }
+
+ @Override
+ public void run(ConnectContext ctx, StmtExecutor executor) throws Exception {
+ if (policyType == PolicyTypeEnum.DATA_MASK) {
+ if (user != null) {
+ user.analyze();
+ }
+ // check auth
+ if (!Env.getCurrentEnv().getAccessManager().checkGlobalPriv(ConnectContext.get(), PrivPredicate.ADMIN)) {
+ ErrorReport.reportAnalysisException(ErrorCode.ERR_SPECIFIC_ACCESS_DENIED_ERROR, "ADMIN");
+ }
+ ShowResultSet showResultSet = Env.getCurrentEnv().getPolicyMgr().showPolicy(user, roleName, policyType);
+ executor.sendResultSet(showResultSet);
+ return;
+ }
+ ctx.getSessionVariable().enableFallbackToOriginalPlannerOnce();
Review Comment:
ditto
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature] support data mask in doris internal auth [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38301:
URL: https://github.com/apache/doris/pull/38301#issuecomment-2309219506
<details>
<summary>TPC-DS: <b>Total hot run time: 191225 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a489ed918b712b14923cea181ab90f7a8b885a9b, data reload: false
query1 1229 881 845 845
query2 6317 1899 1821 1821
query3 10581 3968 3979 3968
query4 59315 23653 23136 23136
query5 5349 496 498 496
query6 420 161 169 161
query7 5769 295 305 295
query8 293 215 201 201
query9 9008 2511 2486 2486
query10 494 280 263 263
query11 18191 15027 15172 15027
query12 158 109 108 108
query13 1577 401 390 390
query14 11234 7437 7058 7058
query15 248 165 171 165
query16 7708 450 461 450
query17 1140 587 599 587
query18 2136 287 289 287
query19 286 146 142 142
query20 119 111 110 110
query21 207 106 110 106
query22 4462 4345 4476 4345
query23 34125 33869 33415 33415
query24 5957 2856 2807 2807
query25 527 387 376 376
query26 683 158 155 155
query27 1777 291 279 279
query28 3756 2072 2049 2049
query29 701 424 404 404
query30 236 154 145 145
query31 923 756 741 741
query32 83 56 54 54
query33 438 286 282 282
query34 843 470 475 470
query35 806 710 713 710
query36 1049 946 937 937
query37 144 87 90 87
query38 4035 3947 3907 3907
query39 1438 1391 1393 1391
query40 198 115 115 115
query41 45 45 44 44
query42 120 97 97 97
query43 492 469 462 462
query44 1111 763 783 763
query45 195 166 165 165
query46 1082 759 735 735
query47 1840 1774 1805 1774
query48 383 298 294 294
query49 755 450 485 450
query50 818 428 415 415
query51 7210 6986 7031 6986
query52 100 88 86 86
query53 250 181 177 177
query54 560 450 451 450
query55 76 77 77 77
query56 278 257 268 257
query57 1208 1056 1049 1049
query58 223 240 231 231
query59 2913 2684 2778 2684
query60 300 260 277 260
query61 101 101 101 101
query62 725 631 652 631
query63 215 186 187 186
query64 4071 2224 1754 1754
query65 3183 3141 3164 3141
query66 664 333 333 333
query67 15236 15299 15373 15299
query68 2973 581 579 579
query69 406 288 285 285
query70 1152 1036 1063 1036
query71 361 279 279 279
query72 2617 2079 2032 2032
query73 704 328 326 326
query74 9163 8820 8719 8719
query75 3354 2707 2666 2666
query76 1416 959 997 959
query77 555 325 321 321
query78 10311 9080 8976 8976
query79 1024 524 534 524
query80 674 529 506 506
query81 444 227 231 227
query82 294 137 134 134
query83 167 148 147 147
query84 261 76 80 76
query85 668 281 279 279
query86 318 283 297 283
query87 4320 4267 4268 4267
query88 2949 2420 2467 2420
query89 385 289 287 287
query90 1911 200 199 199
query91 122 101 97 97
query92 62 51 51 51
query93 1050 538 551 538
query94 688 284 278 278
query95 304 267 265 265
query96 588 271 276 271
query97 3208 3061 3059 3059
query98 222 205 197 197
query99 1503 1311 1270 1270
Total cold run time: 302288 ms
Total hot run time: 191225 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature] support data mask in doris internal auth [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38301:
URL: https://github.com/apache/doris/pull/38301#issuecomment-2301300472
<details>
<summary>ClickBench: <b>Total hot run time: 31.57 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 97f81fcc30a6d38dfc9786c36ba7a755683492dd, data reload: false
query1 0.05 0.04 0.04
query2 0.08 0.04 0.04
query3 0.23 0.05 0.06
query4 1.65 0.08 0.08
query5 0.53 0.49 0.49
query6 1.14 0.72 0.74
query7 0.02 0.02 0.02
query8 0.06 0.04 0.05
query9 0.55 0.49 0.50
query10 0.55 0.55 0.55
query11 0.16 0.12 0.12
query12 0.15 0.13 0.13
query13 0.63 0.59 0.58
query14 0.78 0.81 0.78
query15 0.84 0.82 0.84
query16 0.38 0.36 0.38
query17 0.96 1.06 0.98
query18 0.22 0.20 0.21
query19 1.81 1.71 1.76
query20 0.01 0.01 0.02
query21 15.42 0.66 0.65
query22 4.63 6.30 2.47
query23 18.27 1.44 1.32
query24 2.22 0.23 0.22
query25 0.15 0.10 0.09
query26 0.28 0.19 0.18
query27 0.08 0.09 0.08
query28 13.17 1.04 1.02
query29 12.66 3.29 3.28
query30 0.44 0.26 0.24
query31 2.81 0.41 0.40
query32 3.24 0.49 0.48
query33 2.96 2.98 2.93
query34 16.92 4.39 4.36
query35 4.48 4.42 4.42
query36 0.67 0.48 0.50
query37 0.22 0.18 0.18
query38 0.18 0.16 0.16
query39 0.07 0.06 0.06
query40 0.19 0.15 0.16
query41 0.12 0.07 0.06
query42 0.08 0.07 0.07
query43 0.08 0.07 0.06
Total cold run time: 110.14 s
Total hot run time: 31.57 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](privilege) support data mask in doris internal auth [doris]
Posted by "buu-nguyen (via GitHub)" <gi...@apache.org>.
buu-nguyen commented on PR #38301:
URL: https://github.com/apache/doris/pull/38301#issuecomment-3717240741
Bump this up
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](privilege) support data mask in doris internal auth [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38301:
URL: https://github.com/apache/doris/pull/38301#issuecomment-3696311659
<details>
<summary>TPC-H: <b>Total hot run time: 35671 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3fbd57010ef630762780e3570c5f35efbdd25451, data reload: false
------ Round 1 ----------------------------------
q1 17589 4246 4097 4097
q2 2011 349 234 234
q3 10182 1274 743 743
q4 10210 842 323 323
q5 7557 2142 1922 1922
q6 196 174 140 140
q7 948 795 703 703
q8 9311 1416 1145 1145
q9 6712 5188 5189 5188
q10 6791 1800 1415 1415
q11 506 295 303 295
q12 684 749 621 621
q13 17786 3810 3076 3076
q14 285 306 275 275
q15 583 510 505 505
q16 700 680 638 638
q17 711 715 625 625
q18 7891 7582 7981 7582
q19 1304 1056 683 683
q20 453 396 258 258
q21 4558 4167 4189 4167
q22 1142 1062 1036 1036
Total cold run time: 108110 ms
Total hot run time: 35671 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4369 4297 4245 4245
q2 344 412 327 327
q3 2265 3000 2458 2458
q4 1383 1861 1367 1367
q5 4535 4302 4321 4302
q6 222 172 159 159
q7 1995 1883 1796 1796
q8 2508 2387 2361 2361
q9 7072 7022 7231 7022
q10 2343 2577 2125 2125
q11 524 451 442 442
q12 711 682 594 594
q13 3357 3792 3054 3054
q14 272 273 263 263
q15 523 500 479 479
q16 602 653 616 616
q17 1075 1244 1349 1244
q18 7276 7164 7255 7164
q19 847 836 863 836
q20 1888 1969 1800 1800
q21 4590 4285 4072 4072
q22 1117 1049 973 973
Total cold run time: 49818 ms
Total hot run time: 47699 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](privilege) support data mask in doris internal auth [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #38301:
URL: https://github.com/apache/doris/pull/38301#issuecomment-3707772432
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature] support data mask in doris internal auth [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38301:
URL: https://github.com/apache/doris/pull/38301#issuecomment-2297861438
<details>
<summary>ClickBench: <b>Total hot run time: 31.64 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4ce03c77373d16e926520b11f85aa2cf1640b82a, data reload: false
query1 0.06 0.05 0.04
query2 0.08 0.05 0.05
query3 0.24 0.06 0.07
query4 1.64 0.09 0.09
query5 0.51 0.49 0.49
query6 1.14 0.73 0.72
query7 0.02 0.02 0.01
query8 0.06 0.05 0.05
query9 0.54 0.49 0.48
query10 0.56 0.53 0.54
query11 0.16 0.12 0.12
query12 0.16 0.13 0.14
query13 0.62 0.60 0.60
query14 0.78 0.79 0.77
query15 0.86 0.82 0.84
query16 0.38 0.39 0.38
query17 1.01 1.00 1.03
query18 0.21 0.21 0.22
query19 1.82 1.78 1.87
query20 0.02 0.01 0.01
query21 15.40 0.66 0.65
query22 4.12 6.97 2.32
query23 18.30 1.47 1.29
query24 2.08 0.24 0.22
query25 0.15 0.08 0.09
query26 0.28 0.19 0.18
query27 0.08 0.09 0.08
query28 13.21 1.04 1.01
query29 12.62 3.35 3.30
query30 0.44 0.26 0.25
query31 2.83 0.41 0.41
query32 3.24 0.50 0.50
query33 2.99 2.94 2.99
query34 16.99 4.37 4.35
query35 4.46 4.49 4.48
query36 0.68 0.49 0.51
query37 0.21 0.18 0.17
query38 0.17 0.17 0.17
query39 0.06 0.06 0.06
query40 0.18 0.15 0.15
query41 0.12 0.07 0.07
query42 0.08 0.07 0.07
query43 0.06 0.07 0.06
Total cold run time: 109.62 s
Total hot run time: 31.64 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature] support data mask in doris internal auth [doris]
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow commented on code in PR #38301:
URL: https://github.com/apache/doris/pull/38301#discussion_r1722924118
##########
fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4:
##########
@@ -151,6 +152,10 @@ supportedCreateStatement
AS type=(RESTRICTIVE | PERMISSIVE)
TO (user=userIdentify | ROLE roleName=identifier)
USING LEFT_PAREN booleanExpression RIGHT_PAREN #createRowPolicy
+ | CREATE DATA MASK POLICY (IF NOT EXISTS)? name=identifier
+ ON column=multipartIdentifier
+ TO (user=userIdentify | ROLE roleName=identifier)
+ USING dataMaskType=identifier #createDataMaskPolicy
Review Comment:
```suggestion
| CREATE DATA MASK POLICY (IF NOT EXISTS)? name=identifier
ON column=multipartIdentifier
TO (user=userIdentify | ROLE roleName=identifier)
USING dataMaskType=identifier #createDataMaskPolicy
```
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/DropPolicyCommand.java:
##########
@@ -0,0 +1,79 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.plans.commands;
+
+import org.apache.doris.analysis.DropPolicyStmt;
+import org.apache.doris.analysis.StmtType;
+import org.apache.doris.catalog.Env;
+import org.apache.doris.common.ErrorCode;
+import org.apache.doris.common.ErrorReport;
+import org.apache.doris.mysql.privilege.PrivPredicate;
+import org.apache.doris.nereids.exceptions.AnalysisException;
+import org.apache.doris.nereids.trees.plans.PlanType;
+import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
+import org.apache.doris.policy.PolicyTypeEnum;
+import org.apache.doris.qe.ConnectContext;
+import org.apache.doris.qe.StmtExecutor;
+
+/**
+ * drop policy command
+ */
+public class DropPolicyCommand extends Command implements ForwardWithSync {
+
+ private final PolicyTypeEnum policyType;
+ private final String policyName;
+ private final boolean ifExists;
+
+ /**
+ * constructor for drop command
+ */
+ public DropPolicyCommand(PolicyTypeEnum policyType, String policyName, boolean ifExists) {
+ super(PlanType.DROP_POLICY_COMMAND);
+ this.policyType = policyType;
+ this.policyName = policyName;
+ this.ifExists = ifExists;
+ }
+
+ @Override
+ public <R, C> R accept(PlanVisitor<R, C> visitor, C context) {
+ return visitor.visitDropPolicyCommand(this, context);
+ }
+
+ @Override
+ public void run(ConnectContext ctx, StmtExecutor executor) throws Exception {
+ if (policyType == PolicyTypeEnum.DATA_MASK) {
+ DropPolicyStmt dropPolicyStmt = new DropPolicyStmt(PolicyTypeEnum.DATA_MASK, ifExists, policyName,
Review Comment:
do not reuse DropPolicyStmt
##########
fe/fe-core/src/main/java/org/apache/doris/analysis/ColumnName.java:
##########
@@ -0,0 +1,162 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/apache/impala/blob/branch-2.9.0/fe/src/main/java/org/apache/impala/TableName.java
+// and modified by Doris
+
+package org.apache.doris.analysis;
+
+import org.apache.doris.catalog.Env;
+import org.apache.doris.common.AnalysisException;
+import org.apache.doris.common.ErrorCode;
+import org.apache.doris.common.ErrorReport;
+import org.apache.doris.common.io.Writable;
+import org.apache.doris.datasource.InternalCatalog;
+
+import com.google.common.base.Strings;
+import com.google.gson.annotations.SerializedName;
+
+import java.io.DataOutput;
+import java.io.IOException;
+import java.util.Objects;
+import java.util.stream.Stream;
+
+public class ColumnName implements Writable {
Review Comment:
```suggestion
public class ColumnName {
```
##########
fe/fe-core/src/main/java/org/apache/doris/policy/DorisDataMaskPolicy.java:
##########
@@ -0,0 +1,221 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.policy;
+
+import org.apache.doris.analysis.UserIdentity;
+import org.apache.doris.catalog.Column;
+import org.apache.doris.catalog.ScalarType;
+import org.apache.doris.catalog.Type;
+import org.apache.doris.common.AnalysisException;
+import org.apache.doris.mysql.privilege.DataMaskPolicy;
+import org.apache.doris.nereids.parser.NereidsParser;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.literal.StringLiteral;
+import org.apache.doris.qe.ShowResultSetMetaData;
+
+import com.google.common.collect.Lists;
+import com.google.gson.annotations.SerializedName;
+import lombok.Data;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Locale;
+import java.util.Objects;
+
+@Data
+public class DorisDataMaskPolicy extends Policy implements DataMaskPolicy {
+
+ public static final ShowResultSetMetaData DATA_MASK_META_DATA =
+ ShowResultSetMetaData.builder()
+ .addColumn(new Column("PolicyName", ScalarType.createVarchar(100)))
+ .addColumn(new Column("CatalogName", ScalarType.createVarchar(100)))
+ .addColumn(new Column("DbName", ScalarType.createVarchar(100)))
+ .addColumn(new Column("TableName", ScalarType.createVarchar(100)))
+ .addColumn(new Column("ColumnName", ScalarType.createVarchar(100)))
+ .addColumn(new Column("DataMaskType", ScalarType.createVarchar(20)))
+ .addColumn(new Column("DataMaskDef", ScalarType.createVarchar(65535)))
Review Comment:
varchar length max is 65533
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/CreatePolicyCommand.java:
##########
@@ -80,6 +88,27 @@ public <R, C> R accept(PlanVisitor<R, C> visitor, C context) {
@Override
public void run(ConnectContext ctx, StmtExecutor executor) throws Exception {
+ if (policyType == PolicyTypeEnum.DATA_MASK) {
+ ColumnName columnName = new ColumnName(nameParts.get(0), nameParts.get(1), nameParts.get(2),
+ nameParts.get(3));
+ CreatePolicyStmt policyStmt = new CreatePolicyStmt(PolicyTypeEnum.DATA_MASK, ifNotExists, policyName,
Review Comment:
if data mask policy is a new feature, please do not reuse CreatePolicyStmt to create it in Nereids, because we want to remove all xxxStmt in future.
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/util/RelationUtil.java:
##########
@@ -79,6 +79,53 @@ public static List<String> getQualifierName(ConnectContext context, List<String>
}
}
+ /**
+ * get qualifier column name
+ */
+ public static List<String> getQualifierColumnName(ConnectContext context, List<String> nameParts) {
+ switch (nameParts.size()) {
+ case 1: { // col
+ throw new IllegalStateException("must specify a table.");
Review Comment:
throw Nereids' AnalysisException
##########
fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4:
##########
@@ -161,6 +166,11 @@ supportedAlterStatement
supportedDropStatement
: DROP CATALOG RECYCLE BIN WHERE idType=STRING_LITERAL EQ id=INTEGER_VALUE #dropCatalogRecycleBin
+ | DROP DATA MASK POLICY (IF EXISTS)? name=identifier #dropDataMaskPolicy
Review Comment:
```suggestion
: DROP CATALOG RECYCLE BIN WHERE idType=STRING_LITERAL EQ id=INTEGER_VALUE #dropCatalogRecycleBin
| DROP DATA MASK POLICY (IF EXISTS)? name=identifier
#dropDataMaskPolicy
```
##########
fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/InternalAccessController.java:
##########
@@ -78,7 +79,9 @@ public boolean checkCloudPriv(UserIdentity currentUser, String resourceName,
@Override
public Optional<DataMaskPolicy> evalDataMaskPolicy(UserIdentity currentUser, String ctl, String db, String tbl,
String col) {
- return Optional.empty();
+ DorisDataMaskPolicy dataMaskPolicy = Env.getCurrentEnv().getPolicyMgr().getDataMaskPolicy(ctl, db, tbl, col,
+ currentUser);
+ return dataMaskPolicy == null ? Optional.empty() : Optional.of(dataMaskPolicy);
Review Comment:
```suggestion
return Optional.ofNullable(dataMaskPolicy);
```
##########
fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisLexer.g4:
##########
@@ -371,6 +371,7 @@ LOGICAL: 'LOGICAL';
LOW_PRIORITY: 'LOW_PRIORITY';
MANUAL: 'MANUAL';
MAP: 'MAP';
+MASK: 'MASK';
Review Comment:
should add to non-reserved key word list in DorisParser.g4
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ShowPolicyCommand.java:
##########
@@ -0,0 +1,75 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.plans.commands;
+
+import org.apache.doris.analysis.ShowPolicyStmt;
+import org.apache.doris.analysis.UserIdentity;
+import org.apache.doris.catalog.Env;
+import org.apache.doris.common.ErrorCode;
+import org.apache.doris.common.ErrorReport;
+import org.apache.doris.mysql.privilege.PrivPredicate;
+import org.apache.doris.nereids.exceptions.AnalysisException;
+import org.apache.doris.nereids.trees.plans.PlanType;
+import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
+import org.apache.doris.policy.PolicyTypeEnum;
+import org.apache.doris.qe.ConnectContext;
+import org.apache.doris.qe.ShowResultSet;
+import org.apache.doris.qe.StmtExecutor;
+
+/**
+ * show policy
+ */
+public class ShowPolicyCommand extends Command implements NoForward {
+
+ private final PolicyTypeEnum policyType;
+
+ private final UserIdentity user;
+
+ private final String roleName;
+
+ public ShowPolicyCommand(PolicyTypeEnum policyType, UserIdentity user, String roleName) {
+ super(PlanType.SHOW_POLICY_COMMAND);
+ this.policyType = policyType;
+ this.user = user;
+ this.roleName = roleName;
+ }
+
+ @Override
+ public <R, C> R accept(PlanVisitor<R, C> visitor, C context) {
+ return visitor.visitShowPolicyCommand(this, context);
+ }
+
+ @Override
+ public void run(ConnectContext ctx, StmtExecutor executor) throws Exception {
+ if (policyType == PolicyTypeEnum.DATA_MASK) {
+ ShowPolicyStmt showPolicyStmt = new ShowPolicyStmt(PolicyTypeEnum.DATA_MASK, user, roleName);
Review Comment:
do not reuse ShowPolicyStmt
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature] support data mask in doris internal auth [doris]
Posted by "qzsee (via GitHub)" <gi...@apache.org>.
qzsee commented on PR #38301:
URL: https://github.com/apache/doris/pull/38301#issuecomment-2297808306
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](privilege) support data mask in doris internal auth [doris]
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow commented on PR #38301:
URL: https://github.com/apache/doris/pull/38301#issuecomment-2360575932
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](privilege) support data mask in doris internal auth [doris]
Posted by "qzsee (via GitHub)" <gi...@apache.org>.
qzsee commented on PR #38301:
URL: https://github.com/apache/doris/pull/38301#issuecomment-2367366229
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature] support data mask in doris internal auth [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38301:
URL: https://github.com/apache/doris/pull/38301#issuecomment-2309209196
<details>
<summary>TPC-H: <b>Total hot run time: 37621 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a489ed918b712b14923cea181ab90f7a8b885a9b, data reload: false
------ Round 1 ----------------------------------
q1 17620 4414 4315 4315
q2 2020 176 177 176
q3 11754 981 1112 981
q4 10515 679 700 679
q5 7752 2795 2795 2795
q6 223 134 135 134
q7 955 605 606 605
q8 9300 2036 2068 2036
q9 7286 6514 6531 6514
q10 6991 2173 2202 2173
q11 465 242 242 242
q12 389 213 222 213
q13 17755 3024 2985 2985
q14 285 245 227 227
q15 525 481 486 481
q16 489 394 385 385
q17 954 643 690 643
q18 7257 6779 6923 6779
q19 1389 1141 1034 1034
q20 666 341 314 314
q21 4473 2991 2910 2910
q22 1099 1020 1000 1000
Total cold run time: 110162 ms
Total hot run time: 37621 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4435 4268 4307 4268
q2 367 283 282 282
q3 2850 2676 2656 2656
q4 1955 1638 1669 1638
q5 5512 5680 5634 5634
q6 225 130 135 130
q7 2278 1771 1771 1771
q8 3267 3408 3399 3399
q9 8804 8745 8727 8727
q10 3579 3360 3377 3360
q11 615 524 510 510
q12 810 672 668 668
q13 14675 3118 3189 3118
q14 320 318 280 280
q15 532 493 476 476
q16 490 431 430 430
q17 1833 1568 1532 1532
q18 8200 7856 7828 7828
q19 1746 1553 1675 1553
q20 2148 1923 1899 1899
q21 5764 5497 5463 5463
q22 1154 1034 1022 1022
Total cold run time: 71559 ms
Total hot run time: 56644 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](privilege) support data mask in doris internal auth [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #38301:
URL: https://github.com/apache/doris/pull/38301#issuecomment-2317000015
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](privilege) support data mask in doris internal auth [doris]
Posted by "dqz123 (via GitHub)" <gi...@apache.org>.
dqz123 commented on PR #38301:
URL: https://github.com/apache/doris/pull/38301#issuecomment-3696224363
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature] support data mask in doris internal auth [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38301:
URL: https://github.com/apache/doris/pull/38301#issuecomment-2247162335
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website).
See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature] support data mask in doris internal auth [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38301:
URL: https://github.com/apache/doris/pull/38301#issuecomment-2301269897
<details>
<summary>TPC-H: <b>Total hot run time: 37850 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 97f81fcc30a6d38dfc9786c36ba7a755683492dd, data reload: false
------ Round 1 ----------------------------------
q1 17858 4427 4255 4255
q2 2058 211 216 211
q3 11764 1035 1078 1035
q4 10539 720 711 711
q5 7793 2839 2761 2761
q6 268 157 161 157
q7 1016 652 677 652
q8 9577 2079 2102 2079
q9 8736 6526 6554 6526
q10 7048 2256 2198 2198
q11 502 272 276 272
q12 433 263 263 263
q13 17776 3031 3014 3014
q14 305 262 250 250
q15 550 532 510 510
q16 523 413 410 410
q17 994 676 755 676
q18 7300 6812 6746 6746
q19 6476 1117 1125 1117
q20 667 359 343 343
q21 4299 2868 2660 2660
q22 1105 1014 1004 1004
Total cold run time: 117587 ms
Total hot run time: 37850 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4636 4287 4251 4251
q2 393 300 294 294
q3 2878 2648 2709 2648
q4 1931 1729 1669 1669
q5 5690 5677 5719 5677
q6 243 155 159 155
q7 2196 1794 1793 1793
q8 3292 3534 3459 3459
q9 8797 8708 8762 8708
q10 3595 3349 3323 3323
q11 680 544 530 530
q12 817 691 670 670
q13 17229 3220 3106 3106
q14 317 296 302 296
q15 557 513 512 512
q16 503 463 468 463
q17 1835 1567 1519 1519
q18 8280 7842 7728 7728
q19 8569 1703 1582 1582
q20 2206 1899 1923 1899
q21 13608 5300 5433 5300
q22 1184 1038 1065 1038
Total cold run time: 89436 ms
Total hot run time: 56620 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature] support data mask in doris internal auth [doris]
Posted by "qzsee (via GitHub)" <gi...@apache.org>.
qzsee commented on PR #38301:
URL: https://github.com/apache/doris/pull/38301#issuecomment-2304009090
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](privilege) support data mask in doris internal auth [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38301:
URL: https://github.com/apache/doris/pull/38301#issuecomment-2315662832
<details>
<summary>TPC-H: <b>Total hot run time: 38045 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 63e67853eeb83a5749b675fd264a663eb58b5fe6, data reload: false
------ Round 1 ----------------------------------
q1 17598 4433 4263 4263
q2 2011 176 174 174
q3 11873 976 1113 976
q4 10514 757 701 701
q5 7758 2833 2814 2814
q6 229 142 136 136
q7 959 613 599 599
q8 9327 2102 2090 2090
q9 7196 6553 6520 6520
q10 6991 2279 2180 2180
q11 506 239 240 239
q12 393 232 218 218
q13 18742 3071 3023 3023
q14 280 241 233 233
q15 525 497 488 488
q16 576 505 522 505
q17 976 663 684 663
q18 7242 6953 6875 6875
q19 1397 984 1006 984
q20 683 338 341 338
q21 3848 3026 2999 2999
q22 1140 1061 1027 1027
Total cold run time: 110764 ms
Total hot run time: 38045 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4321 4301 4247 4247
q2 389 271 274 271
q3 2864 2701 2684 2684
q4 1930 1653 1625 1625
q5 5611 5672 5721 5672
q6 242 137 141 137
q7 2243 1857 1875 1857
q8 3298 3421 3457 3421
q9 8837 8866 8774 8774
q10 3640 3415 3370 3370
q11 613 499 504 499
q12 850 666 658 658
q13 15837 3125 3255 3125
q14 323 307 278 278
q15 534 492 485 485
q16 625 574 592 574
q17 1847 1549 1538 1538
q18 8135 7964 7921 7921
q19 1746 1561 1583 1561
q20 2187 1905 1913 1905
q21 5887 5593 5394 5394
q22 1137 1062 1071 1062
Total cold run time: 73096 ms
Total hot run time: 57058 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](privilege) support data mask in doris internal auth [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38301:
URL: https://github.com/apache/doris/pull/38301#issuecomment-2315701710
<details>
<summary>ClickBench: <b>Total hot run time: 31.8 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 63e67853eeb83a5749b675fd264a663eb58b5fe6, data reload: false
query1 0.05 0.05 0.04
query2 0.07 0.04 0.04
query3 0.23 0.05 0.04
query4 1.68 0.06 0.07
query5 0.52 0.50 0.51
query6 1.13 0.76 0.71
query7 0.02 0.01 0.02
query8 0.05 0.04 0.04
query9 0.56 0.50 0.50
query10 0.54 0.52 0.56
query11 0.15 0.11 0.12
query12 0.15 0.13 0.13
query13 0.61 0.61 0.58
query14 2.03 2.07 2.04
query15 0.84 0.82 0.81
query16 0.37 0.38 0.37
query17 1.00 1.05 1.06
query18 0.21 0.20 0.21
query19 1.86 1.74 1.81
query20 0.01 0.01 0.02
query21 15.39 0.67 0.66
query22 4.69 7.46 1.65
query23 18.30 1.37 1.32
query24 2.16 0.22 0.24
query25 0.15 0.08 0.09
query26 0.27 0.18 0.18
query27 0.07 0.08 0.08
query28 13.18 1.04 1.01
query29 12.60 3.29 3.29
query30 0.25 0.05 0.05
query31 2.90 0.40 0.39
query32 3.27 0.47 0.47
query33 2.97 2.98 2.97
query34 17.01 4.43 4.40
query35 4.46 4.54 4.46
query36 0.66 0.48 0.47
query37 0.19 0.16 0.15
query38 0.16 0.15 0.15
query39 0.05 0.04 0.04
query40 0.15 0.13 0.12
query41 0.10 0.05 0.05
query42 0.06 0.05 0.05
query43 0.04 0.04 0.04
Total cold run time: 111.16 s
Total hot run time: 31.8 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](privilege) support data mask in doris internal auth [doris]
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow commented on PR #38301:
URL: https://github.com/apache/doris/pull/38301#issuecomment-2316998985
run cloud_p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature] support data mask in doris internal auth [doris]
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow commented on code in PR #38301:
URL: https://github.com/apache/doris/pull/38301#discussion_r1726475940
##########
fe/fe-core/src/main/cup/sql_parser.cup:
##########
@@ -498,6 +498,7 @@ terminal String
KW_MATCH_PHRASE_PREFIX,
KW_MATCH_REGEXP,
KW_MATCH_PHRASE_EDGE,
+ KW_MASK,
Review Comment:
add mask to non-reserved key word list
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ShowPolicyCommand.java:
##########
@@ -0,0 +1,73 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.plans.commands;
+
+import org.apache.doris.analysis.UserIdentity;
+import org.apache.doris.catalog.Env;
+import org.apache.doris.common.ErrorCode;
+import org.apache.doris.common.ErrorReport;
+import org.apache.doris.mysql.privilege.PrivPredicate;
+import org.apache.doris.nereids.exceptions.AnalysisException;
+import org.apache.doris.nereids.trees.plans.PlanType;
+import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
+import org.apache.doris.policy.PolicyTypeEnum;
+import org.apache.doris.qe.ConnectContext;
+import org.apache.doris.qe.ShowResultSet;
+import org.apache.doris.qe.StmtExecutor;
+
+/**
+ * show policy
+ */
+public class ShowPolicyCommand extends Command implements NoForward {
+
+ private final PolicyTypeEnum policyType;
+
+ private final UserIdentity user;
+
+ private final String roleName;
+
+ public ShowPolicyCommand(PolicyTypeEnum policyType, UserIdentity user, String roleName) {
+ super(PlanType.SHOW_POLICY_COMMAND);
+ this.policyType = policyType;
+ this.user = user;
+ this.roleName = roleName;
+ }
+
+ @Override
+ public <R, C> R accept(PlanVisitor<R, C> visitor, C context) {
+ return visitor.visitShowPolicyCommand(this, context);
+ }
+
+ @Override
+ public void run(ConnectContext ctx, StmtExecutor executor) throws Exception {
+ if (policyType == PolicyTypeEnum.DATA_MASK) {
+ if (user != null) {
+ user.analyze();
+ }
+ // check auth
+ if (!Env.getCurrentEnv().getAccessManager().checkGlobalPriv(ConnectContext.get(), PrivPredicate.ADMIN)) {
+ ErrorReport.reportAnalysisException(ErrorCode.ERR_SPECIFIC_ACCESS_DENIED_ERROR, "ADMIN");
+ }
+ ShowResultSet showResultSet = Env.getCurrentEnv().getPolicyMgr().showPolicy(user, roleName, policyType);
+ executor.sendResultSet(showResultSet);
+ return;
+ }
+ ctx.getSessionVariable().enableFallbackToOriginalPlannerOnce();
+ throw new AnalysisException("Not support show policy command in Nereids now");
Review Comment:
throw new MustFallbackException
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/DropPolicyCommand.java:
##########
@@ -0,0 +1,78 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.plans.commands;
+
+import org.apache.doris.analysis.StmtType;
+import org.apache.doris.catalog.Env;
+import org.apache.doris.common.ErrorCode;
+import org.apache.doris.common.ErrorReport;
+import org.apache.doris.mysql.privilege.PrivPredicate;
+import org.apache.doris.nereids.exceptions.AnalysisException;
+import org.apache.doris.nereids.trees.plans.PlanType;
+import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
+import org.apache.doris.policy.DropPolicyLog;
+import org.apache.doris.policy.PolicyTypeEnum;
+import org.apache.doris.qe.ConnectContext;
+import org.apache.doris.qe.StmtExecutor;
+
+/**
+ * drop policy command
+ */
+public class DropPolicyCommand extends Command implements ForwardWithSync {
+
+ private final PolicyTypeEnum policyType;
+ private final String policyName;
+ private final boolean ifExists;
+
+ /**
+ * constructor for drop command
+ */
+ public DropPolicyCommand(PolicyTypeEnum policyType, String policyName, boolean ifExists) {
+ super(PlanType.DROP_POLICY_COMMAND);
+ this.policyType = policyType;
+ this.policyName = policyName;
+ this.ifExists = ifExists;
+ }
+
+ @Override
+ public <R, C> R accept(PlanVisitor<R, C> visitor, C context) {
+ return visitor.visitDropPolicyCommand(this, context);
+ }
+
+ @Override
+ public void run(ConnectContext ctx, StmtExecutor executor) throws Exception {
+ if (policyType == PolicyTypeEnum.DATA_MASK) {
+ // check auth
+ if (!Env.getCurrentEnv().getAccessManager()
+ .checkGlobalPriv(ConnectContext.get(), PrivPredicate.GRANT)) {
+ ErrorReport.reportAnalysisException(ErrorCode.ERR_SPECIFIC_ACCESS_DENIED_ERROR,
+ PrivPredicate.GRANT.getPrivs().toString());
+ }
+ DropPolicyLog dropPolicyLog = new DropPolicyLog(policyType, policyName);
+ Env.getCurrentEnv().getPolicyMgr().dropPolicy(dropPolicyLog, ifExists);
+ return;
+ }
+ ctx.getSessionVariable().enableFallbackToOriginalPlannerOnce();
+ throw new AnalysisException("Not support drop policy command in Nereids now");
Review Comment:
use MustFallbackException
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature] support data mask in doris internal auth [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38301:
URL: https://github.com/apache/doris/pull/38301#issuecomment-2311446325
<details>
<summary>ClickBench: <b>Total hot run time: 30.24 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a489ed918b712b14923cea181ab90f7a8b885a9b, data reload: false
query1 0.05 0.04 0.04
query2 0.08 0.04 0.04
query3 0.22 0.04 0.05
query4 1.68 0.08 0.08
query5 0.50 0.49 0.49
query6 1.13 0.72 0.72
query7 0.02 0.02 0.01
query8 0.05 0.04 0.04
query9 0.54 0.47 0.49
query10 0.54 0.55 0.52
query11 0.15 0.11 0.11
query12 0.15 0.12 0.12
query13 0.61 0.59 0.58
query14 0.76 0.80 0.79
query15 0.85 0.81 0.82
query16 0.38 0.38 0.38
query17 1.06 1.07 1.04
query18 0.21 0.21 0.21
query19 1.87 1.71 1.76
query20 0.01 0.01 0.01
query21 15.42 0.67 0.66
query22 4.96 7.63 1.54
query23 18.21 1.29 1.26
query24 2.07 0.22 0.22
query25 0.15 0.08 0.08
query26 0.28 0.19 0.18
query27 0.08 0.07 0.08
query28 13.30 1.02 0.99
query29 12.65 3.32 3.35
query30 0.24 0.05 0.06
query31 2.88 0.41 0.40
query32 3.24 0.50 0.48
query33 2.95 2.99 3.00
query34 16.78 4.37 4.35
query35 4.46 4.38 4.39
query36 0.67 0.50 0.47
query37 0.19 0.15 0.16
query38 0.17 0.15 0.15
query39 0.04 0.03 0.03
query40 0.16 0.12 0.13
query41 0.10 0.05 0.05
query42 0.06 0.05 0.05
query43 0.04 0.04 0.04
Total cold run time: 109.96 s
Total hot run time: 30.24 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](privilege) support data mask in doris internal auth [doris]
Posted by "dqz123 (via GitHub)" <gi...@apache.org>.
dqz123 commented on PR #38301:
URL: https://github.com/apache/doris/pull/38301#issuecomment-2662938211
run p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](privilege) support data mask in doris internal auth [doris]
Posted by "qzsee (via GitHub)" <gi...@apache.org>.
qzsee commented on PR #38301:
URL: https://github.com/apache/doris/pull/38301#issuecomment-2315572464
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature] support data mask in doris internal auth [doris]
Posted by "qzsee (via GitHub)" <gi...@apache.org>.
qzsee commented on PR #38301:
URL: https://github.com/apache/doris/pull/38301#issuecomment-2311394489
run FE UT
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](privilege) support data mask in doris internal auth [doris]
Posted by "qzsee (via GitHub)" <gi...@apache.org>.
qzsee commented on PR #38301:
URL: https://github.com/apache/doris/pull/38301#issuecomment-2371046151
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](privilege) support data mask in doris internal auth [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38301:
URL: https://github.com/apache/doris/pull/38301#issuecomment-2371248679
<details>
<summary>ClickBench: <b>Total hot run time: 33.06 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c38ecbfce784ec4c7da85e972b4973d7778e5c8c, data reload: false
query1 0.05 0.04 0.04
query2 0.06 0.03 0.03
query3 0.23 0.07 0.07
query4 1.64 0.10 0.10
query5 0.52 0.51 0.52
query6 1.13 0.72 0.72
query7 0.02 0.01 0.01
query8 0.04 0.03 0.03
query9 0.54 0.49 0.50
query10 0.55 0.58 0.55
query11 0.14 0.09 0.10
query12 0.14 0.11 0.10
query13 0.62 0.59 0.59
query14 3.10 2.92 2.96
query15 0.88 0.82 0.83
query16 0.41 0.38 0.41
query17 1.00 1.06 0.99
query18 0.19 0.20 0.20
query19 1.89 1.89 1.98
query20 0.01 0.00 0.00
query21 15.35 0.59 0.57
query22 2.38 2.63 2.74
query23 17.35 1.05 0.78
query24 2.64 0.33 0.88
query25 0.11 0.22 0.06
query26 0.43 0.14 0.13
query27 0.05 0.04 0.05
query28 11.74 1.09 1.07
query29 12.55 3.32 3.29
query30 0.24 0.06 0.06
query31 2.89 0.40 0.38
query32 3.26 0.46 0.46
query33 2.97 3.02 3.07
query34 16.75 4.50 4.44
query35 4.47 4.50 4.56
query36 0.66 0.47 0.48
query37 0.08 0.06 0.06
query38 0.05 0.03 0.04
query39 0.03 0.02 0.02
query40 0.15 0.12 0.12
query41 0.08 0.02 0.02
query42 0.03 0.02 0.02
query43 0.04 0.03 0.03
Total cold run time: 107.46 s
Total hot run time: 33.06 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature] support data mask in doris internal auth [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38301:
URL: https://github.com/apache/doris/pull/38301#issuecomment-2295551989
<details>
<summary>ClickBench: <b>Total hot run time: 31.78 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2b0383da1930218dfb731890284147b74ced8996, data reload: false
query1 0.05 0.04 0.04
query2 0.09 0.04 0.04
query3 0.24 0.05 0.06
query4 1.66 0.08 0.08
query5 0.48 0.48 0.49
query6 1.14 0.73 0.73
query7 0.02 0.02 0.02
query8 0.05 0.05 0.05
query9 0.56 0.49 0.49
query10 0.54 0.55 0.56
query11 0.16 0.12 0.12
query12 0.15 0.12 0.12
query13 0.63 0.60 0.60
query14 0.78 0.78 0.78
query15 0.87 0.82 0.82
query16 0.36 0.36 0.39
query17 0.98 0.99 0.96
query18 0.23 0.22 0.21
query19 1.91 1.82 1.74
query20 0.01 0.02 0.01
query21 15.40 0.66 0.66
query22 4.78 6.36 2.47
query23 18.34 1.36 1.40
query24 2.10 0.22 0.23
query25 0.15 0.09 0.08
query26 0.32 0.23 0.24
query27 0.47 0.23 0.22
query28 13.30 1.01 1.00
query29 12.64 3.30 3.31
query30 0.39 0.21 0.20
query31 2.82 0.41 0.41
query32 3.23 0.49 0.49
query33 2.96 2.96 2.98
query34 17.16 4.31 4.35
query35 4.40 4.44 4.41
query36 0.66 0.48 0.48
query37 0.22 0.18 0.17
query38 0.17 0.16 0.17
query39 0.06 0.06 0.06
query40 0.18 0.16 0.15
query41 0.12 0.06 0.07
query42 0.08 0.08 0.07
query43 0.07 0.07 0.06
Total cold run time: 110.93 s
Total hot run time: 31.78 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature] support data mask in doris internal auth [doris]
Posted by "qzsee (via GitHub)" <gi...@apache.org>.
qzsee commented on PR #38301:
URL: https://github.com/apache/doris/pull/38301#issuecomment-2293677642
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature] support data mask in doris internal auth [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38301:
URL: https://github.com/apache/doris/pull/38301#issuecomment-2297847394
<details>
<summary>TPC-H: <b>Total hot run time: 38002 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4ce03c77373d16e926520b11f85aa2cf1640b82a, data reload: false
------ Round 1 ----------------------------------
q1 17959 4397 4307 4307
q2 2060 212 211 211
q3 11860 977 1120 977
q4 10540 756 728 728
q5 7786 2876 2806 2806
q6 265 164 161 161
q7 1018 661 642 642
q8 9597 2097 2106 2097
q9 8806 6581 6577 6577
q10 7086 2243 2210 2210
q11 517 281 266 266
q12 427 268 257 257
q13 18836 2988 2969 2969
q14 324 263 245 245
q15 549 529 531 529
q16 516 430 410 410
q17 981 640 748 640
q18 7358 6729 6777 6729
q19 6394 1092 1036 1036
q20 719 343 345 343
q21 3961 3169 2821 2821
q22 1144 1041 1061 1041
Total cold run time: 118703 ms
Total hot run time: 38002 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4514 4276 4282 4276
q2 410 309 300 300
q3 2835 2664 2724 2664
q4 1982 1714 1671 1671
q5 5643 5731 5717 5717
q6 242 156 169 156
q7 2212 1796 1801 1796
q8 3316 3519 3476 3476
q9 8778 8785 8756 8756
q10 3614 3343 3343 3343
q11 628 524 535 524
q12 855 668 653 653
q13 17057 3144 3061 3061
q14 315 305 313 305
q15 587 536 539 536
q16 520 494 470 470
q17 1853 1565 1537 1537
q18 8189 7932 7685 7685
q19 9494 1648 1726 1648
q20 2184 1920 1886 1886
q21 12325 5236 5355 5236
q22 1213 1070 1106 1070
Total cold run time: 88766 ms
Total hot run time: 56766 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature] support data mask in doris internal auth [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38301:
URL: https://github.com/apache/doris/pull/38301#issuecomment-2297857242
<details>
<summary>TPC-DS: <b>Total hot run time: 196600 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4ce03c77373d16e926520b11f85aa2cf1640b82a, data reload: false
query1 1312 909 889 889
query2 6578 2006 1874 1874
query3 10736 4011 3828 3828
query4 59272 24302 23346 23346
query5 6037 732 734 732
query6 555 211 212 211
query7 6452 332 331 331
query8 548 444 436 436
query9 9319 2553 2525 2525
query10 626 354 346 346
query11 18773 15135 15519 15135
query12 207 130 130 130
query13 1711 428 456 428
query14 12504 7610 7291 7291
query15 270 203 198 198
query16 7929 532 514 514
query17 1234 658 625 625
query18 2297 374 348 348
query19 290 197 174 174
query20 160 138 147 138
query21 245 147 164 147
query22 4757 4318 4489 4318
query23 34308 33746 33810 33746
query24 5701 3015 3072 3015
query25 586 441 434 434
query26 732 187 188 187
query27 1840 315 307 307
query28 3978 2204 2126 2126
query29 768 445 459 445
query30 222 179 181 179
query31 1031 837 853 837
query32 121 77 80 77
query33 566 347 343 343
query34 897 501 501 501
query35 863 784 774 774
query36 1115 957 965 957
query37 164 111 102 102
query38 3995 3905 3877 3877
query39 1680 1530 1455 1455
query40 241 160 165 160
query41 140 137 136 136
query42 137 116 119 116
query43 556 507 511 507
query44 1128 800 806 800
query45 227 196 196 196
query46 1133 780 811 780
query47 1908 1863 1878 1863
query48 416 338 331 331
query49 912 582 574 574
query50 863 484 469 469
query51 6845 6778 6805 6778
query52 117 113 113 113
query53 304 228 233 228
query54 614 507 501 501
query55 90 89 91 89
query56 331 325 310 310
query57 1239 1114 1138 1114
query58 303 337 323 323
query59 3076 3008 2850 2850
query60 351 332 335 332
query61 148 143 151 143
query62 795 687 712 687
query63 282 227 228 227
query64 4420 2371 1870 1870
query65 3231 3240 3182 3182
query66 1012 673 669 669
query67 15272 14920 14886 14886
query68 6477 591 586 586
query69 715 428 328 328
query70 1244 1182 1189 1182
query71 541 322 319 319
query72 7045 2305 2075 2075
query73 822 355 366 355
query74 9271 8840 8926 8840
query75 4266 2763 2749 2749
query76 4005 1066 1050 1050
query77 886 443 449 443
query78 9934 8986 9039 8986
query79 6747 566 574 566
query80 1238 631 631 631
query81 605 266 266 266
query82 872 168 167 167
query83 425 217 229 217
query84 321 99 106 99
query85 1048 354 358 354
query86 436 336 303 303
query87 4454 4275 4294 4275
query88 4626 2490 2513 2490
query89 440 324 330 324
query90 2035 232 237 232
query91 155 129 130 129
query92 88 77 76 76
query93 5544 555 549 549
query94 865 286 343 286
query95 395 297 299 297
query96 612 286 284 284
query97 3274 3079 3095 3079
query98 252 230 232 230
query99 1587 1295 1273 1273
Total cold run time: 337695 ms
Total hot run time: 196600 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature] support data mask in doris internal auth [doris]
Posted by "zddr (via GitHub)" <gi...@apache.org>.
zddr commented on PR #38301:
URL: https://github.com/apache/doris/pull/38301#issuecomment-2303568750
we will support `custom` type like ranger?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](privilege) support data mask in doris internal auth [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38301:
URL: https://github.com/apache/doris/pull/38301#issuecomment-3696341224
<details>
<summary>ClickBench: <b>Total hot run time: 26.72 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3fbd57010ef630762780e3570c5f35efbdd25451, data reload: false
query1 0.06 0.05 0.05
query2 0.11 0.05 0.04
query3 0.25 0.09 0.09
query4 1.60 0.12 0.11
query5 0.28 0.24 0.25
query6 1.15 0.66 0.65
query7 0.03 0.02 0.03
query8 0.06 0.04 0.04
query9 0.56 0.50 0.49
query10 0.55 0.55 0.53
query11 0.16 0.11 0.11
query12 0.16 0.12 0.13
query13 0.62 0.61 0.61
query14 0.98 0.99 0.97
query15 0.81 0.79 0.80
query16 0.39 0.40 0.40
query17 1.01 1.03 1.02
query18 0.23 0.22 0.21
query19 1.98 1.86 1.76
query20 0.01 0.02 0.01
query21 15.46 0.30 0.15
query22 4.76 0.05 0.05
query23 15.97 0.29 0.11
query24 1.95 0.66 0.21
query25 0.09 0.06 0.08
query26 0.14 0.12 0.14
query27 0.06 0.06 0.04
query28 5.44 1.04 0.87
query29 12.62 3.98 3.16
query30 0.28 0.14 0.12
query31 2.83 0.63 0.37
query32 3.23 0.56 0.48
query33 3.04 2.97 3.03
query34 16.44 5.09 4.41
query35 4.45 4.54 4.51
query36 0.66 0.49 0.50
query37 0.10 0.07 0.07
query38 0.08 0.04 0.04
query39 0.05 0.03 0.03
query40 0.17 0.15 0.12
query41 0.08 0.03 0.03
query42 0.04 0.02 0.03
query43 0.05 0.03 0.03
Total cold run time: 98.99 s
Total hot run time: 26.72 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](privilege) support data mask in doris internal auth [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #38301: [Feature](privilege) support data mask in doris internal auth
URL: https://github.com/apache/doris/pull/38301
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](privilege) support data mask in doris internal auth [doris]
Posted by "qzsee (via GitHub)" <gi...@apache.org>.
qzsee commented on PR #38301:
URL: https://github.com/apache/doris/pull/38301#issuecomment-2372869195
run cloud_p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature] support data mask in doris internal auth [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38301:
URL: https://github.com/apache/doris/pull/38301#issuecomment-2304102100
<details>
<summary>TPC-DS: <b>Total hot run time: 191612 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 738658c3a73a9ce4dee98920146dc3dfe67cd382, data reload: false
query1 1241 885 879 879
query2 6294 1914 1920 1914
query3 10573 3985 3922 3922
query4 59912 25191 23317 23317
query5 5446 531 507 507
query6 408 163 166 163
query7 5753 308 306 306
query8 301 205 206 205
query9 8967 2461 2436 2436
query10 516 263 267 263
query11 18020 14912 15140 14912
query12 161 101 100 100
query13 1586 387 407 387
query14 10783 7330 7845 7330
query15 243 175 179 175
query16 7096 469 492 469
query17 1170 566 577 566
query18 1245 310 307 307
query19 290 158 150 150
query20 122 112 112 112
query21 203 107 100 100
query22 4604 4411 4468 4411
query23 34411 33983 33270 33270
query24 6009 2884 2844 2844
query25 547 395 399 395
query26 693 163 160 160
query27 1777 290 280 280
query28 3961 2030 2019 2019
query29 670 430 435 430
query30 238 152 151 151
query31 944 777 749 749
query32 85 56 62 56
query33 457 297 287 287
query34 880 495 480 480
query35 842 734 729 729
query36 1091 945 934 934
query37 139 85 90 85
query38 4016 3867 3855 3855
query39 1475 1399 1401 1399
query40 205 120 124 120
query41 48 49 46 46
query42 114 96 98 96
query43 501 466 463 463
query44 1134 740 758 740
query45 198 167 165 165
query46 1096 796 731 731
query47 1895 1768 1828 1768
query48 365 297 299 297
query49 752 421 437 421
query50 822 421 424 421
query51 7213 7105 6961 6961
query52 104 87 87 87
query53 252 181 179 179
query54 575 455 447 447
query55 76 76 78 76
query56 274 268 262 262
query57 1184 1041 1070 1041
query58 220 219 266 219
query59 2922 2837 2947 2837
query60 293 275 274 274
query61 103 116 101 101
query62 751 672 668 668
query63 225 188 183 183
query64 4207 2281 1764 1764
query65 3248 3193 3156 3156
query66 629 346 335 335
query67 15420 15305 15135 15135
query68 3078 588 589 588
query69 405 282 275 275
query70 1128 1147 1028 1028
query71 336 278 281 278
query72 2741 2135 2032 2032
query73 720 324 375 324
query74 9217 8832 8801 8801
query75 3337 2625 2681 2625
query76 1552 1008 1012 1008
query77 549 314 313 313
query78 9644 9006 9100 9006
query79 1029 546 537 537
query80 686 508 512 508
query81 454 230 228 228
query82 296 140 136 136
query83 172 151 148 148
query84 254 82 75 75
query85 676 301 298 298
query86 313 297 329 297
query87 4500 4320 4192 4192
query88 3083 2377 2349 2349
query89 380 291 283 283
query90 1996 193 200 193
query91 124 125 106 106
query92 61 52 53 52
query93 1064 533 542 533
query94 671 307 300 300
query95 335 264 259 259
query96 587 266 265 265
query97 3240 3054 3091 3054
query98 209 194 204 194
query99 1529 1281 1280 1280
Total cold run time: 302361 ms
Total hot run time: 191612 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature] support data mask in doris internal auth [doris]
Posted by "qzsee (via GitHub)" <gi...@apache.org>.
qzsee commented on PR #38301:
URL: https://github.com/apache/doris/pull/38301#issuecomment-2298042208
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature] support data mask in doris internal auth [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38301:
URL: https://github.com/apache/doris/pull/38301#issuecomment-2300485332
<details>
<summary>TPC-H: <b>Total hot run time: 38256 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ab922a92600decac2ca8b14f7729e015620e877d, data reload: false
------ Round 1 ----------------------------------
q1 17900 4325 4275 4275
q2 2058 217 225 217
q3 11772 1031 1221 1031
q4 10528 725 706 706
q5 7778 2829 2857 2829
q6 266 157 158 157
q7 1011 671 658 658
q8 9578 2105 2124 2105
q9 8754 6555 6548 6548
q10 7091 2245 2238 2238
q11 489 276 271 271
q12 439 266 281 266
q13 19005 2985 3019 2985
q14 298 251 253 251
q15 559 524 524 524
q16 526 418 402 402
q17 990 671 744 671
q18 7307 6796 6704 6704
q19 6316 1121 1102 1102
q20 691 351 334 334
q21 4102 2933 2974 2933
q22 1122 1059 1049 1049
Total cold run time: 118580 ms
Total hot run time: 38256 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4514 4282 4227 4227
q2 398 312 290 290
q3 2854 2655 2701 2655
q4 2006 1701 1712 1701
q5 5621 5699 5661 5661
q6 243 155 165 155
q7 2198 1805 1755 1755
q8 3290 3548 3492 3492
q9 8750 8915 8702 8702
q10 3550 3364 3276 3276
q11 650 535 524 524
q12 862 680 638 638
q13 15879 3210 3198 3198
q14 329 295 291 291
q15 561 520 522 520
q16 490 444 441 441
q17 1858 1517 1552 1517
q18 8676 8169 7948 7948
q19 1760 1564 1499 1499
q20 2173 1927 1903 1903
q21 13046 5422 5361 5361
q22 1173 1091 1069 1069
Total cold run time: 80881 ms
Total hot run time: 56823 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature] support data mask in doris internal auth [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #38301:
URL: https://github.com/apache/doris/pull/38301#issuecomment-2301543235
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature] support data mask in doris internal auth [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38301:
URL: https://github.com/apache/doris/pull/38301#issuecomment-2311431365
<details>
<summary>TPC-H: <b>Total hot run time: 37763 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a489ed918b712b14923cea181ab90f7a8b885a9b, data reload: false
------ Round 1 ----------------------------------
q1 17633 4370 4286 4286
q2 2013 185 171 171
q3 11698 953 1099 953
q4 10496 736 735 735
q5 7758 2819 2801 2801
q6 226 135 140 135
q7 965 610 607 607
q8 9320 2066 2062 2062
q9 7226 6534 6541 6534
q10 7015 2218 2134 2134
q11 444 252 244 244
q12 388 219 228 219
q13 18521 3082 3036 3036
q14 286 238 228 228
q15 514 487 489 487
q16 508 403 393 393
q17 992 671 725 671
q18 7318 6748 6807 6748
q19 1383 1017 1075 1017
q20 717 332 323 323
q21 3910 3038 2963 2963
q22 1122 1016 1018 1016
Total cold run time: 110453 ms
Total hot run time: 37763 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4284 4285 4293 4285
q2 375 261 275 261
q3 2871 2626 2675 2626
q4 1928 1668 1593 1593
q5 5493 5676 5720 5676
q6 235 139 134 134
q7 2264 1868 1811 1811
q8 3336 3387 3427 3387
q9 8863 8808 8858 8808
q10 3619 3392 3367 3367
q11 619 509 497 497
q12 821 673 696 673
q13 15683 3204 3171 3171
q14 327 307 289 289
q15 526 480 476 476
q16 502 454 462 454
q17 1840 1561 1514 1514
q18 8099 7802 7823 7802
q19 1727 1482 1639 1482
q20 2139 1907 1893 1893
q21 5674 5497 5309 5309
q22 1128 1053 1045 1045
Total cold run time: 72353 ms
Total hot run time: 56553 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](privilege) support data mask in doris internal auth [doris]
Posted by "qzsee (via GitHub)" <gi...@apache.org>.
qzsee commented on PR #38301:
URL: https://github.com/apache/doris/pull/38301#issuecomment-2347839024
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](privilege) support data mask in doris internal auth [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38301:
URL: https://github.com/apache/doris/pull/38301#issuecomment-2662596509
<details>
<summary>TPC-H: <b>Total hot run time: 31662 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8f011cd6290513cf910b91ba1fd59477ca98c7db, data reload: false
------ Round 1 ----------------------------------
q1 17616 5332 5104 5104
q2 2041 307 173 173
q3 10391 1342 703 703
q4 10214 1015 553 553
q5 7533 2400 2349 2349
q6 189 167 135 135
q7 888 747 619 619
q8 9309 1267 1130 1130
q9 4964 4573 4816 4573
q10 6820 2311 1881 1881
q11 478 277 262 262
q12 360 353 223 223
q13 17779 3717 3076 3076
q14 222 226 213 213
q15 497 457 463 457
q16 627 610 589 589
q17 588 870 349 349
q18 6743 6202 6272 6202
q19 1082 942 539 539
q20 308 328 197 197
q21 2853 2163 2026 2026
q22 354 319 309 309
Total cold run time: 101856 ms
Total hot run time: 31662 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5164 5120 5196 5120
q2 228 332 232 232
q3 2156 2638 2298 2298
q4 1451 1884 1419 1419
q5 4229 4135 4144 4135
q6 204 163 126 126
q7 1869 1807 1646 1646
q8 2680 2578 2471 2471
q9 7258 7051 7069 7051
q10 2980 3219 2791 2791
q11 568 504 493 493
q12 699 743 617 617
q13 3572 3859 3322 3322
q14 271 307 279 279
q15 506 448 482 448
q16 635 670 632 632
q17 1151 1630 1322 1322
q18 7544 7246 7381 7246
q19 844 890 877 877
q20 1997 2021 1876 1876
q21 5476 5052 4823 4823
q22 617 574 576 574
Total cold run time: 52099 ms
Total hot run time: 49798 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](privilege) support data mask in doris internal auth [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38301:
URL: https://github.com/apache/doris/pull/38301#issuecomment-2662626358
<details>
<summary>TPC-DS: <b>Total hot run time: 183244 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8f011cd6290513cf910b91ba1fd59477ca98c7db, data reload: false
query1 952 370 385 370
query2 6525 1884 1832 1832
query3 6794 219 216 216
query4 26106 23901 22973 22973
query5 4352 670 518 518
query6 306 245 193 193
query7 4599 517 290 290
query8 308 251 249 249
query9 8625 2494 2471 2471
query10 472 330 258 258
query11 15169 15132 14902 14902
query12 153 113 108 108
query13 1657 529 407 407
query14 9657 6810 6259 6259
query15 228 196 183 183
query16 7787 645 483 483
query17 1213 734 559 559
query18 1999 408 318 318
query19 202 187 161 161
query20 129 117 110 110
query21 205 128 104 104
query22 4312 4306 4580 4306
query23 34378 33400 32953 32953
query24 7781 2383 2369 2369
query25 515 459 386 386
query26 1223 270 151 151
query27 2111 503 334 334
query28 3887 2376 2361 2361
query29 711 535 427 427
query30 233 184 153 153
query31 928 820 767 767
query32 80 63 62 62
query33 560 343 339 339
query34 783 851 509 509
query35 786 835 769 769
query36 965 1019 897 897
query37 128 97 73 73
query38 4240 4201 4118 4118
query39 1600 1445 1381 1381
query40 203 116 106 106
query41 56 55 52 52
query42 135 107 100 100
query43 501 496 480 480
query44 1321 799 776 776
query45 179 172 158 158
query46 853 1044 650 650
query47 1760 1776 1730 1730
query48 377 426 297 297
query49 785 491 417 417
query50 673 750 423 423
query51 4211 4210 4155 4155
query52 120 111 96 96
query53 231 261 182 182
query54 500 482 410 410
query55 81 78 80 78
query56 270 277 248 248
query57 1119 1172 1058 1058
query58 246 242 249 242
query59 2659 2792 2490 2490
query60 290 263 261 261
query61 119 149 120 120
query62 798 727 662 662
query63 236 200 193 193
query64 4257 985 660 660
query65 3282 3143 3155 3143
query66 1053 395 300 300
query67 15898 15572 15406 15406
query68 7997 770 499 499
query69 502 297 263 263
query70 1149 1164 1121 1121
query71 397 305 262 262
query72 5774 3556 3814 3556
query73 743 746 348 348
query74 9252 8942 9016 8942
query75 3185 3174 2685 2685
query76 3287 1173 778 778
query77 492 410 288 288
query78 10043 10008 9355 9355
query79 2022 790 579 579
query80 665 532 450 450
query81 507 270 241 241
query82 202 129 100 100
query83 174 172 150 150
query84 241 89 70 70
query85 750 353 323 323
query86 372 319 295 295
query87 4458 4488 4365 4365
query88 3260 2191 2172 2172
query89 383 310 281 281
query90 2003 189 190 189
query91 207 137 106 106
query92 76 59 54 54
query93 2121 1000 577 577
query94 681 403 292 292
query95 357 268 255 255
query96 490 555 267 267
query97 2740 2858 2732 2732
query98 237 208 205 205
query99 1351 1409 1232 1232
Total cold run time: 270990 ms
Total hot run time: 183244 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](privilege) support data mask in doris internal auth [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38301:
URL: https://github.com/apache/doris/pull/38301#issuecomment-3696332219
<details>
<summary>TPC-DS: <b>Total hot run time: 174203 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3fbd57010ef630762780e3570c5f35efbdd25451, data reload: false
query5 4753 587 426 426
query6 330 222 226 222
query7 4204 457 274 274
query8 337 232 230 230
query9 8757 2596 2602 2596
query10 499 381 321 321
query11 15202 15123 14894 14894
query12 165 113 110 110
query13 1253 483 382 382
query14 6164 2951 2752 2752
query14_1 2599 2667 2608 2608
query15 206 197 177 177
query16 1002 469 462 462
query17 1086 702 582 582
query18 2569 434 354 354
query19 230 220 195 195
query20 128 120 117 117
query21 220 137 124 124
query22 3914 4016 3801 3801
query23 16010 15671 15168 15168
query23_1 15281 15512 15414 15414
query24 7385 1589 1187 1187
query24_1 1207 1228 1232 1228
query25 574 489 451 451
query26 1252 267 151 151
query27 2764 446 288 288
query28 4594 2176 2166 2166
query29 785 524 418 418
query30 307 242 209 209
query31 789 625 545 545
query32 76 70 63 63
query33 530 329 280 280
query34 921 885 520 520
query35 763 778 690 690
query36 819 883 836 836
query37 123 87 71 71
query38 2699 2673 2663 2663
query39 773 755 735 735
query39_1 715 725 703 703
query40 223 130 112 112
query41 70 63 65 63
query42 107 107 102 102
query43 454 463 407 407
query44 1323 758 756 756
query45 189 184 174 174
query46 862 970 613 613
query47 1448 1447 1396 1396
query48 338 318 247 247
query49 615 453 319 319
query50 646 280 201 201
query51 3761 3760 3950 3760
query52 107 103 95 95
query53 290 316 271 271
query54 289 259 247 247
query55 78 78 71 71
query56 293 284 288 284
query57 962 1026 935 935
query58 251 255 243 243
query59 2055 2095 1960 1960
query60 314 311 304 304
query61 162 167 156 156
query62 383 352 317 317
query63 293 261 271 261
query64 5007 1281 976 976
query65 3759 3676 3708 3676
query66 1412 443 314 314
query67 16093 15113 14900 14900
query68 4706 989 742 742
query69 497 346 307 307
query70 1055 936 938 936
query71 362 297 262 262
query72 6299 4959 5135 4959
query73 699 587 318 318
query74 8813 8739 8706 8706
query75 2886 2864 2528 2528
query76 3869 1061 650 650
query77 522 360 269 269
query78 9576 9884 9163 9163
query79 1486 927 612 612
query80 683 608 497 497
query81 512 258 234 234
query82 230 148 109 109
query83 254 269 241 241
query84 260 122 101 101
query85 946 521 473 473
query86 389 308 322 308
query87 3025 2870 2792 2792
query88 3309 2313 2310 2310
query89 395 358 333 333
query90 2029 156 149 149
query91 172 167 142 142
query92 74 65 63 63
query93 1126 940 570 570
query94 557 336 277 277
query95 569 382 305 305
query96 602 477 212 212
query97 2324 2379 2264 2264
query98 229 197 198 197
query99 554 568 494 494
Total cold run time: 252126 ms
Total hot run time: 174203 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](privilege) support data mask in doris internal auth [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #38301: [Feature](privilege) support data mask in doris internal auth
URL: https://github.com/apache/doris/pull/38301
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](privilege) support data mask in doris internal auth [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #38301:
URL: https://github.com/apache/doris/pull/38301#issuecomment-5433169782
We're closing this PR because it hasn't been updated in a while.
This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and feel free a maintainer to remove the Stale tag!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org