You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "morningman (via GitHub)" <gi...@apache.org> on 2026/03/30 04:03:17 UTC
[PR] [refactor](fe) Phase 0: Prerequisite decoupling for filesystem SPI [doris]
morningman opened a new pull request, #61859:
URL: https://github.com/apache/doris/pull/61859
### What problem does this PR solve?
Issue Number: N/A
Problem Summary: Before splitting filesystem implementations into independent Maven modules (Phase 3), several compile-time couplings must be eliminated. This commit completes all Phase 0 prerequisite decoupling tasks:
- P0.1: Introduce FsStorageType enum in fe-foundation (zero-dep module) to replace StorageBackend.StorageType (Thrift-generated) in PersistentFileSystem. Add FsStorageTypeAdapter for bidirectional Thrift conversion. Update all subclasses and callers (Repository, BackupJob, RestoreJob, CloudRestoreJob).
- P0.2: Add IOException-based default bridge methods to ObjStorage interface (checkObjectExists, getObjectChecked, putObjectChecked, deleteObjectChecked, deleteObjectsChecked, copyObjectChecked, listObjectsChecked). Add ObjStorageStatusAdapter for Status→IOException conversion. Zero changes to existing implementations.
- P0.3: Decouple SwitchingFileSystem from ExternalMetaCacheMgr via new FileSystemLookup functional interface. FileSystemProviderImpl passes a lambda.
- P0.4: Extract MultipartUploadCapable interface from ObjFileSystem, removing the forced abstract method. S3FileSystem and AzureFileSystem implement it. HMSTransaction now uses instanceof check instead of ObjFileSystem cast.
- P0.5: Introduce FileSystemDescriptor POJO for Repository metadata serialization, replacing direct PersistentFileSystem subclass serialization. Migrate GsonUtils to string-based Class.forName() reflection for legacy format backward compat, removing 7 compile-time imports of concrete filesystem classes.
- P0.6: Add FileSystemSpiProvider interface skeleton in fs/spi/ as the future ServiceLoader contract for Phase 3 module split.
### Release note
None
### Check List (For Author)
- Test: No need to test (pure refactor; all changes are backward compatible; three successful FE builds verified during development)
- Behavior changed: No
- Does this need documentation: No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](fe) Phase 0: Prerequisite decoupling for filesystem SPI [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman closed pull request #61859: [refactor](fe) Phase 0: Prerequisite decoupling for filesystem SPI
URL: https://github.com/apache/doris/pull/61859
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](fe) Phase 0: Prerequisite decoupling for filesystem SPI [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #61859:
URL: https://github.com/apache/doris/pull/61859#issuecomment-4152258845
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](fe) Phase 0: Prerequisite decoupling for filesystem SPI [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #61859:
URL: https://github.com/apache/doris/pull/61859#issuecomment-4152113469
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](fe) Phase 0: Prerequisite decoupling for filesystem SPI [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #61859:
URL: https://github.com/apache/doris/pull/61859#issuecomment-4152388225
<details>
<summary>TPC-DS: <b>Total hot run time: 168948 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ed6b4cfdeb42fa85ba8868bd741b51614473f595, data reload: false
query5 4338 629 499 499
query6 339 231 200 200
query7 4209 451 258 258
query8 336 234 219 219
query9 8709 2735 2675 2675
query10 523 396 337 337
query11 6939 5058 4876 4876
query12 191 129 123 123
query13 1279 455 352 352
query14 5729 3722 3435 3435
query14_1 2853 2920 2830 2830
query15 210 199 176 176
query16 985 486 464 464
query17 902 739 643 643
query18 2427 434 336 336
query19 205 204 183 183
query20 128 123 121 121
query21 202 131 107 107
query22 13285 13891 14696 13891
query23 16533 16342 16266 16266
query23_1 16260 15718 15670 15670
query24 7220 1600 1215 1215
query24_1 1234 1224 1213 1213
query25 555 460 441 441
query26 1275 264 151 151
query27 2768 474 294 294
query28 4533 1818 1816 1816
query29 827 572 480 480
query30 286 225 189 189
query31 1022 938 866 866
query32 82 74 69 69
query33 509 337 287 287
query34 899 859 544 544
query35 639 675 605 605
query36 1110 1129 1001 1001
query37 140 98 76 76
query38 2978 2990 2878 2878
query39 859 836 813 813
query39_1 794 783 801 783
query40 237 147 144 144
query41 64 60 56 56
query42 258 253 252 252
query43 246 258 219 219
query44
query45 204 190 186 186
query46 892 991 601 601
query47 2144 2150 2099 2099
query48 333 322 231 231
query49 639 508 391 391
query50 719 279 225 225
query51 4105 4109 4047 4047
query52 272 282 264 264
query53 302 341 285 285
query54 335 272 265 265
query55 101 92 87 87
query56 322 345 305 305
query57 1927 1707 1701 1701
query58 287 276 267 267
query59 2785 2930 2717 2717
query60 341 330 316 316
query61 152 154 155 154
query62 640 591 544 544
query63 310 276 274 274
query64 5088 1277 1003 1003
query65
query66 1466 473 362 362
query67 24134 24282 24109 24109
query68
query69 420 317 299 299
query70 999 912 937 912
query71 330 312 297 297
query72 2999 2855 2678 2678
query73 535 537 326 326
query74 9610 9539 9356 9356
query75 2875 2757 2461 2461
query76 2325 1053 666 666
query77 372 396 322 322
query78 10920 11084 10478 10478
query79 2984 775 573 573
query80 1763 622 564 564
query81 582 255 219 219
query82 1011 156 124 124
query83 330 267 239 239
query84 311 119 105 105
query85 914 481 436 436
query86 479 302 287 287
query87 3131 3138 3002 3002
query88 3537 2656 2646 2646
query89 422 368 340 340
query90 2032 180 197 180
query91 175 158 139 139
query92 78 78 70 70
query93 1448 815 502 502
query94 644 311 305 305
query95 588 340 323 323
query96 630 514 232 232
query97 2443 2471 2409 2409
query98 231 218 225 218
query99 1022 981 903 903
Total cold run time: 253337 ms
Total hot run time: 168948 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](fe) Phase 0: Prerequisite decoupling for filesystem SPI [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #61859:
URL: https://github.com/apache/doris/pull/61859#issuecomment-4152004670
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](fe) Phase 0: Prerequisite decoupling for filesystem SPI [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #61859:
URL: https://github.com/apache/doris/pull/61859#issuecomment-4152173553
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](fe) Phase 0: Prerequisite decoupling for filesystem SPI [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #61859:
URL: https://github.com/apache/doris/pull/61859#issuecomment-4152350862
<details>
<summary>TPC-H: <b>Total hot run time: 26365 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ed6b4cfdeb42fa85ba8868bd741b51614473f595, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17610 4453 4275 4275
q2 q3 10640 767 517 517
q4 4680 357 248 248
q5 7541 1193 1018 1018
q6 184 172 145 145
q7 776 826 682 682
q8 9395 1481 1341 1341
q9 4880 4652 4627 4627
q10 6323 1904 1664 1664
q11 471 244 246 244
q12 813 573 468 468
q13 18034 2678 1924 1924
q14 228 228 208 208
q15 q16 742 741 664 664
q17 715 839 449 449
q18 5900 5326 5223 5223
q19 1127 977 637 637
q20 587 490 381 381
q21 4524 1844 1412 1412
q22 361 296 238 238
Total cold run time: 95531 ms
Total hot run time: 26365 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4874 4661 4579 4579
q2 q3 3922 4340 3804 3804
q4 861 1193 780 780
q5 4064 4516 4421 4421
q6 201 177 152 152
q7 1770 1673 1575 1575
q8 2515 2729 2568 2568
q9 7597 7536 7295 7295
q10 3822 4069 3625 3625
q11 554 439 410 410
q12 509 582 441 441
q13 2425 3077 2073 2073
q14 293 330 294 294
q15 q16 742 787 718 718
q17 1244 1492 1396 1396
q18 7234 6811 6720 6720
q19 930 940 885 885
q20 2150 2268 2008 2008
q21 3970 3552 3287 3287
q22 447 433 374 374
Total cold run time: 50124 ms
Total hot run time: 47405 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org