You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "Gabriel39 (via GitHub)" <gi...@apache.org> on 2026/10/16 02:40:26 UTC
[PR] [fix](hudi) Decode Spark SQL timestamp partition paths [doris]
Gabriel39 opened a new pull request, #69157:
URL: https://github.com/apache/doris/pull/69157
### What problem does this PR solve?
Spark SQL's Hudi `SqlKeyGenerator` always escapes formatted TIMESTAMP partition values, independently of `hoodie.table.url.encode.partitioning`. Doris currently unescapes partition values only when that general flag is enabled. With the flag absent or false, a physical component such as `event_time=2024-01-01 10%3A00%3A00` reaches the BE as an invalid timestamp instead of `2024-01-01 10:00:00`. This breaks `test_hudi_partition_prune` and `test_hudi_runtime_filter_partition_pruning`.
Carry timestamp partition-column names from the table schema through Hudi handles and apply exactly one Hive-style unescape to those logical values. Use the same interpretation for HMS/Hudi metadata listing, partition pruning, snapshot pin filtering, and COW/MOR snapshot and incremental scan ranges. Preserve physical file paths. Other types still follow the persisted URL-encoding flag, so a STRING containing literal `%3A`, `%253A`, or `+` remains distinct and unchanged when encoding is disabled.
The writer behavior is documented in [Hudi's SqlKeyGenerator](https://github.com/apache/hudi/blob/release-1.0.2/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/command/SqlKeyGenerator.scala).
### Validation
- Confirmed the new metadata regression fails before the fix with expected `10:00:00`, actual `10%3A00%3A00`.
- `mvn -f fe/pom.xml -pl :fe-connector-hudi -am test -Dmaven.build.cache.enabled=false`: 1,007 tests passed, including 270 Hudi tests; zero failures/errors/skips.
- FE Checkstyle/validate and Groovy syntax compilation passed.
- Coverage includes real persisted SDK configurations with missing/false/true encoding flags, mixed timestamp/STRING partitions, single-layer decoding, nulls, HMS and metadata pruning, historical listing, a real local COW commit/write-stat mapping, and MOR native/JNI Thrift values.
- Extended the existing external regression to compare native/JNI timestamp rows under both partition-listing modes while keeping the existing golden assertion and restoring the session setting.
The real Spark writer and full external Doris/HMS/Hudi suites were not run locally; the new external assertion is syntax-checked and remains for CI.
### Release note
Fix reads of Spark SQL Hudi TIMESTAMP partitions whose paths are escaped while generic URL partition encoding is disabled.
### Check List (For Author)
- [x] Unit Test
- [x] Regression test extension
- [x] Behavior changed: timestamp partition values are decoded according to Spark/Hudi writer semantics
- [x] No documentation needed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](hudi) Decode Spark SQL timestamp partition paths [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69157:
URL: https://github.com/apache/doris/pull/69157#issuecomment-6208431606
<details>
<summary>ClickBench: <b>Total hot run time: 24.65 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 13694449f6c647a0dd18c4a3f5e119e02084f9bf, data reload: false
query1 0.01 0.01 0.00
query2 0.10 0.05 0.05
query3 0.26 0.12 0.12
query4 1.61 0.12 0.12
query5 0.24 0.25 0.25
query6 1.18 0.70 0.67
query7 0.04 0.01 0.00
query8 0.05 0.03 0.03
query9 0.40 0.36 0.36
query10 0.57 0.60 0.59
query11 0.20 0.14 0.15
query12 0.18 0.15 0.14
query13 0.50 0.49 0.50
query14 1.03 1.01 1.02
query15 0.66 0.61 0.60
query16 0.35 0.35 0.35
query17 1.16 1.15 1.18
query18 0.22 0.21 0.21
query19 2.15 2.02 2.00
query20 0.02 0.01 0.02
query21 15.52 0.27 0.14
query22 4.40 0.05 0.05
query23 16.27 0.31 0.12
query24 2.98 0.64 0.43
query25 0.12 0.05 0.05
query26 1.46 0.24 0.15
query27 0.04 0.05 0.04
query28 1.88 0.56 0.36
query29 12.48 4.18 3.31
query30 0.28 0.13 0.13
query31 2.77 0.57 0.33
query32 3.22 0.61 0.52
query33 3.22 3.23 3.25
query34 15.46 4.05 3.40
query35 3.33 3.33 3.34
query36 0.54 0.44 0.45
query37 0.10 0.06 0.06
query38 0.05 0.04 0.03
query39 0.04 0.03 0.03
query40 0.18 0.14 0.14
query41 0.08 0.03 0.02
query42 0.04 0.02 0.02
query43 0.04 0.03 0.04
Total cold run time: 95.43 s
Total hot run time: 24.65 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](hudi) Decode Spark SQL timestamp partition paths [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #69157:
URL: https://github.com/apache/doris/pull/69157#issuecomment-6211355667
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](hudi) Decode Spark SQL timestamp partition paths [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69157:
URL: https://github.com/apache/doris/pull/69157#issuecomment-6208267360
<details>
<summary>TPC-H: <b>Total hot run time: 29224 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 13694449f6c647a0dd18c4a3f5e119e02084f9bf, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17576 5582 5572 5572
q2 2134 334 244 244
q3 10237 1390 834 834
q4 4686 471 351 351
q5 7436 787 529 529
q6 192 189 152 152
q7 741 795 604 604
q8 9235 1346 1085 1085
q9 5608 4245 4237 4237
q10 6815 1294 1003 1003
q11 429 264 237 237
q12 630 409 280 280
q13 18084 3834 3063 3063
q14 278 282 240 240
q15 q16 746 727 674 674
q17 1220 753 576 576
q18 7033 6230 6205 6205
q19 1105 956 601 601
q20 391 335 232 232
q21 5237 2248 2364 2248
q22 370 313 257 257
Total cold run time: 100183 ms
Total hot run time: 29224 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 6146 6112 6103 6103
q2 677 565 548 548
q3 5216 5382 5003 5003
q4 1992 2115 1524 1524
q5 5286 5210 6115 5210
q6 263 196 156 156
q7 2346 2118 1962 1962
q8 3322 3086 2803 2803
q9 8149 8144 8358 8144
q10 3747 3659 3312 3312
q11 641 455 401 401
q12 679 733 497 497
q13 3405 3856 3399 3399
q14 336 329 300 300
q15 q16 707 714 674 674
q17 8288 7557 7304 7304
q18 13219 12323 13053 12323
q19 968 852 821 821
q20 2187 2165 1939 1939
q21 5712 4635 4763 4635
q22 492 466 422 422
Total cold run time: 73778 ms
Total hot run time: 67480 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](hudi) Decode Spark SQL timestamp partition paths [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #69157:
URL: https://github.com/apache/doris/pull/69157#issuecomment-6206832306
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](hudi) Decode Spark SQL timestamp partition paths [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 merged PR #69157:
URL: https://github.com/apache/doris/pull/69157
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](hudi) Decode Spark SQL timestamp partition paths [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69157:
URL: https://github.com/apache/doris/pull/69157#issuecomment-6206820291
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://doris.apache.org/community/how-to-contribute/review-and-ci).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](hudi) Decode Spark SQL timestamp partition paths [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #69157:
URL: https://github.com/apache/doris/pull/69157#issuecomment-6206838363
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](hudi) Decode Spark SQL timestamp partition paths [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69157:
URL: https://github.com/apache/doris/pull/69157#issuecomment-6208376524
<details>
<summary>TPC-DS: <b>Total hot run time: 152281 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 13694449f6c647a0dd18c4a3f5e119e02084f9bf, data reload: false
query5 4314 578 456 456
query6 420 206 187 187
query7 4798 443 220 220
query8 330 179 164 164
query9 8762 3953 3933 3933
query10 467 315 259 259
query11 6053 3803 3535 3535
query12 151 98 87 87
query13 1242 435 325 325
query14 6042 3997 3733 3733
query14_1 3531 3505 3527 3505
query15 211 205 184 184
query16 950 467 382 382
query17 881 694 547 547
query18 2418 423 321 321
query19 201 170 143 143
query20 85 85 85 85
query21 220 136 114 114
query22 12910 12948 12740 12740
query23 13553 12896 12493 12493
query23_1 12635 12763 12627 12627
query24 7090 831 470 470
query24_1 478 469 481 469
query25 541 418 381 381
query26 1270 250 134 134
query27 2770 434 267 267
query28 4598 1935 1946 1935
query29 1540 588 441 441
query30 299 225 182 182
query31 1003 919 790 790
query32 140 87 92 87
query33 498 285 235 235
query34 884 843 488 488
query35 746 760 640 640
query36 819 833 735 735
query37 128 99 85 85
query38 1803 1757 1651 1651
query39 800 806 738 738
query39_1 737 734 744 734
query40 216 115 95 95
query41 64 61 63 61
query42 85 80 82 80
query43 366 380 335 335
query44 1292 692 692 692
query45 178 171 163 163
query46 836 974 563 563
query47 3009 3040 2889 2889
query48 298 319 226 226
query49 569 426 293 293
query50 676 268 207 207
query51 10555 10534 10474 10474
query52 76 78 69 69
query53 189 204 153 153
query54 239 191 187 187
query55 70 70 62 62
query56 217 210 208 208
query57 1624 1587 1530 1530
query58 284 275 254 254
query59 2121 2159 2006 2006
query60 325 245 228 228
query61 153 148 150 148
query62 391 342 296 296
query63 199 160 158 158
query64 2740 983 827 827
query65 3274 3244 3218 3218
query66 1779 417 298 298
query67 20393 20062 19973 19973
query68 3219 1013 609 609
query69 387 287 237 237
query70 855 785 770 770
query71 284 234 207 207
query72 2874 2720 2457 2457
query73 539 534 292 292
query74 4629 4451 4299 4299
query75 2400 2309 1933 1933
query76 2389 1034 609 609
query77 354 381 276 276
query78 9671 9755 9233 9233
query79 995 863 516 516
query80 547 459 402 402
query81 525 314 266 266
query82 579 135 115 115
query83 296 196 185 185
query84 312 128 99 99
query85 799 432 377 377
query86 329 232 219 219
query87 2035 1971 1815 1815
query88 3602 2664 2652 2652
query89 338 289 251 251
query90 1937 178 181 178
query91 163 149 127 127
query92 98 90 91 90
query93 1021 1003 576 576
query94 435 288 254 254
query95 573 345 317 317
query96 665 536 231 231
query97 2464 2445 2354 2354
query98 159 150 147 147
query99 777 756 651 651
Total cold run time: 234517 ms
Total hot run time: 152281 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](hudi) Decode Spark SQL timestamp partition paths [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #69157:
URL: https://github.com/apache/doris/pull/69157#issuecomment-6211354943
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org