You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "zy-kkk (via GitHub)" <gi...@apache.org> on 2024/02/20 08:58:43 UTC
[PR] [Enhancement](jdbc catalog) Add security check on driver when creating Jdbc Catalog [doris]
zy-kkk opened a new pull request, #31153:
URL: https://github.com/apache/doris/pull/31153
## Proposed changes
Issue Number: close #xxx
### Enhancements
This PR introduces enhanced security measures for managing and verifying driver packages when creating Catalogs in Doris. Specifically, it adds path management and checksum verification to ensure the security of driver jars specified by the `driver_url`. The key highlights include:
1. For driver packages specified by filename within the `jdbc_drivers/` directory (configured via `fe.conf` and `be.conf`), Doris assumes these are secure and does not perform additional path checks.
2. For driver packages specified through absolute local paths or HTTP URLs, Doris enforces checks against allowed paths configured via the `jdbc_driver_secure_path` FE configuration item. This setting supports multiple paths separated by semicolons. If the `driver_url`'s path does not match any prefix in `jdbc_driver_secure_path`, the creation of the Catalog is denied.
3. A `checksum` parameter can be specified when creating a Catalog, allowing Doris to verify the driver package's integrity post-load. Catalog creation is aborted if verification fails.
### Upgrade Impact
- The default configuration for `jdbc_driver_secure_path` is `*`, indicating that all driver package paths are allowed. This ensures backward compatibility and does not affect existing Catalogs upon upgrade.
- Since the verification is only performed during Catalog creation, existing Catalogs remain unaffected.
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](jdbc catalog) Add security check on driver when creating Jdbc Catalog [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31153:
URL: https://github.com/apache/doris/pull/31153#issuecomment-1953750666
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](jdbc catalog) Add security check on driver when creating Jdbc Catalog [doris]
Posted by "zy-kkk (via GitHub)" <gi...@apache.org>.
zy-kkk commented on PR #31153:
URL: https://github.com/apache/doris/pull/31153#issuecomment-1953761861
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](jdbc catalog) Add security check on driver when creating Jdbc Catalog [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31153:
URL: https://github.com/apache/doris/pull/31153#issuecomment-1953817442
<details>
<summary>TPC-DS: <b>Total hot run time: 177994 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 20a176c26c20d4efd3ef16b5153067671fb0555b, data reload: false
query1 919 350 357 350
query2 6507 1753 1730 1730
query3 6690 212 201 201
query4 23023 21134 21053 21053
query5 4229 379 383 379
query6 266 176 162 162
query7 4607 300 295 295
query8 248 195 202 195
query9 8428 2799 2807 2799
query10 411 247 232 232
query11 15024 14522 14470 14470
query12 143 84 82 82
query13 1673 409 420 409
query14 9082 8109 7642 7642
query15 217 189 190 189
query16 7386 260 244 244
query17 1412 540 547 540
query18 1967 276 260 260
query19 195 145 147 145
query20 84 84 84 84
query21 190 125 124 124
query22 5313 5105 4805 4805
query23 32600 31616 31635 31616
query24 12181 3336 3460 3336
query25 645 369 357 357
query26 1824 149 163 149
query27 2943 311 315 311
query28 6811 1851 1837 1837
query29 1165 621 607 607
query30 277 136 146 136
query31 935 741 757 741
query32 92 59 56 56
query33 721 231 234 231
query34 1030 483 497 483
query35 926 812 811 811
query36 1039 923 910 910
query37 178 62 64 62
query38 3324 3170 3191 3170
query39 1372 1346 1313 1313
query40 285 105 105 105
query41 36 34 35 34
query42 113 99 94 94
query43 466 437 433 433
query44 1083 695 708 695
query45 192 181 178 178
query46 1033 761 763 761
query47 1691 1563 1598 1563
query48 415 360 341 341
query49 1212 301 300 300
query50 790 385 394 385
query51 5310 5177 5192 5177
query52 109 92 94 92
query53 386 287 291 287
query54 287 220 224 220
query55 84 79 77 77
query56 219 210 200 200
query57 1068 951 946 946
query58 223 204 203 203
query59 2156 2254 2153 2153
query60 237 213 216 213
query61 84 80 79 79
query62 586 380 396 380
query63 322 288 282 282
query64 6323 3050 3140 3050
query65 3303 3250 3234 3234
query66 1324 323 323 323
query67 14602 14770 14315 14315
query68 5078 562 551 551
query69 511 365 365 365
query70 1253 1228 1268 1228
query71 380 253 248 248
query72 6249 2793 2598 2598
query73 700 311 308 308
query74 6931 6559 6523 6523
query75 3157 2556 2537 2537
query76 3243 1161 1217 1161
query77 360 234 230 230
query78 9473 8972 8951 8951
query79 959 505 493 493
query80 493 353 352 352
query81 402 203 206 203
query82 234 86 82 82
query83 138 120 125 120
query84 225 78 78 78
query85 767 343 334 334
query86 285 288 302 288
query87 3510 3361 3281 3281
query88 2679 2278 2262 2262
query89 441 344 356 344
query90 2124 165 166 165
query91 151 130 127 127
query92 55 49 52 49
query93 1004 528 511 511
query94 975 176 183 176
query95 468 375 8821 375
query96 583 260 265 260
query97 4409 4304 4265 4265
query98 217 208 195 195
query99 1141 698 751 698
Total cold run time: 268611 ms
Total hot run time: 177994 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](jdbc catalog) Add security check on driver when creating Jdbc Catalog [doris]
Posted by "zy-kkk (via GitHub)" <gi...@apache.org>.
zy-kkk merged PR #31153:
URL: https://github.com/apache/doris/pull/31153
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](jdbc catalog) Add security check on driver when creating Jdbc Catalog [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31153:
URL: https://github.com/apache/doris/pull/31153#issuecomment-1953831799
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit 20a176c26c20d4efd3ef16b5153067671fb0555b with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 13.5 seconds inserted 10000000 Rows, about 740K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](jdbc catalog) Add security check on driver when creating Jdbc Catalog [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31153:
URL: https://github.com/apache/doris/pull/31153#issuecomment-1955804074
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](jdbc catalog) Add security check on driver when creating Jdbc Catalog [doris]
Posted by "zy-kkk (via GitHub)" <gi...@apache.org>.
zy-kkk commented on PR #31153:
URL: https://github.com/apache/doris/pull/31153#issuecomment-1954193905
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](jdbc catalog) Add security check on driver when creating Jdbc Catalog [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31153:
URL: https://github.com/apache/doris/pull/31153#issuecomment-1953797837
<details>
<summary>TPC-H: <b>Total hot run time: 41851 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 20a176c26c20d4efd3ef16b5153067671fb0555b, data reload: false
------ Round 1 ----------------------------------
q1 17589 4968 4889 4889
q2 2035 142 134 134
q3 10904 1050 1038 1038
q4 4975 1048 1030 1030
q5 8394 3346 3366 3346
q6 199 136 134 134
q7 1288 828 802 802
q8 9573 2095 2092 2092
q9 7643 6714 6708 6708
q10 8320 2651 2657 2651
q11 419 221 222 221
q12 818 323 350 323
q13 17997 3685 3663 3663
q14 290 254 254 254
q15 590 550 494 494
q16 479 413 429 413
q17 938 869 884 869
q18 7355 6669 6654 6654
q19 1539 1500 1505 1500
q20 578 379 333 333
q21 6590 3962 4004 3962
q22 874 343 341 341
Total cold run time: 109387 ms
Total hot run time: 41851 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4875 4847 4854 4847
q2 292 182 183 182
q3 3610 3572 3577 3572
q4 2541 2509 2528 2509
q5 5772 5759 5771 5759
q6 211 129 126 126
q7 2250 1675 1658 1658
q8 3019 3070 3119 3070
q9 8736 8656 8730 8656
q10 6902 4239 4230 4230
q11 500 377 359 359
q12 760 545 539 539
q13 4225 3425 3415 3415
q14 255 235 243 235
q15 629 550 502 502
q16 487 430 456 430
q17 1671 1606 1608 1606
q18 8461 7761 7603 7603
q19 1638 1635 1628 1628
q20 2115 1849 1835 1835
q21 6592 6155 6172 6155
q22 561 510 494 494
Total cold run time: 66102 ms
Total hot run time: 59410 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](jdbc catalog) Add security check on driver when creating Jdbc Catalog [doris]
Posted by "zy-kkk (via GitHub)" <gi...@apache.org>.
zy-kkk commented on PR #31153:
URL: https://github.com/apache/doris/pull/31153#issuecomment-1953752753
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](jdbc catalog) Add security check on driver when creating Jdbc Catalog [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31153:
URL: https://github.com/apache/doris/pull/31153#issuecomment-1953826382
<details>
<summary>ClickBench: <b>Total hot run time: 30.48 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 20a176c26c20d4efd3ef16b5153067671fb0555b, data reload: false
query1 0.03 0.02 0.02
query2 0.06 0.02 0.02
query3 0.23 0.07 0.08
query4 1.63 0.08 0.08
query5 0.49 0.48 0.48
query6 1.38 0.61 0.62
query7 0.02 0.01 0.02
query8 0.04 0.02 0.03
query9 0.52 0.44 0.44
query10 0.48 0.49 0.49
query11 0.12 0.10 0.09
query12 0.12 0.10 0.10
query13 0.58 0.59 0.58
query14 0.76 0.79 0.77
query15 0.82 0.80 0.80
query16 0.32 0.32 0.32
query17 0.89 0.92 0.89
query18 0.19 0.18 0.15
query19 1.75 1.69 1.67
query20 0.01 0.00 0.01
query21 15.43 0.66 0.58
query22 2.95 4.19 2.04
query23 17.41 1.09 0.97
query24 2.02 0.23 0.31
query25 0.65 0.06 0.05
query26 0.16 0.15 0.14
query27 0.06 0.04 0.06
query28 12.38 0.83 0.83
query29 12.59 3.29 3.34
query30 0.52 0.47 0.50
query31 2.77 0.37 0.37
query32 3.34 0.48 0.48
query33 3.15 3.14 3.15
query34 15.35 4.53 4.54
query35 4.51 4.50 4.49
query36 1.08 0.94 0.96
query37 0.07 0.05 0.06
query38 0.04 0.03 0.03
query39 0.02 0.02 0.02
query40 0.16 0.15 0.14
query41 0.07 0.02 0.01
query42 0.02 0.01 0.02
query43 0.02 0.02 0.02
Total cold run time: 105.21 s
Total hot run time: 30.48 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](jdbc catalog) Add security check on driver when creating Jdbc Catalog [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31153:
URL: https://github.com/apache/doris/pull/31153#issuecomment-1955804117
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org