You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "dataroaring (via GitHub)" <gi...@apache.org> on 2026/03/27 02:02:03 UTC
[PR] [fix](azure) Fix incorrect modification timestamps in AzureObjStorage [doris]
dataroaring opened a new pull request, #61790:
URL: https://github.com/apache/doris/pull/61790
## Summary
Fix incorrect modification timestamps in `AzureObjStorage` across three code paths:
- **`globList` LIST path**: Used `getLastModified().getSecond()` which returns the second-of-minute (0–59), not an epoch timestamp
- **`globListByGetProperties` HEAD path**: Used `toEpochSecond()` which returns epoch seconds — off by 1000× from S3's `toEpochMilli()`
- **`listFiles` path**: Same `getSecond()` bug as the glob path
All three are fixed to use `.toInstant().toEpochMilli()`, consistent with `S3ObjStorage` which uses `Instant.toEpochMilli()`.
Found via review of #61775 (cherry-pick of #60414).
## Test plan
- [ ] Verify Azure blob listing returns correct modification timestamps in epoch milliseconds
- [ ] Confirm consistency between S3 and Azure storage backends for `RemoteFile.modifiedTime`
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](azure) Fix incorrect modification timestamps in AzureObjStorage [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #61790:
URL: https://github.com/apache/doris/pull/61790#issuecomment-4140201364
<details>
<summary>TPC-DS: <b>Total hot run time: 168707 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ebb7ca45c66b753583a6c74bc31f3e65a143f201, data reload: false
query5 4341 654 525 525
query6 342 234 203 203
query7 4215 469 274 274
query8 334 244 220 220
query9 8698 2680 2673 2673
query10 531 398 339 339
query11 6995 5110 4846 4846
query12 183 129 124 124
query13 1281 481 361 361
query14 5785 3679 3444 3444
query14_1 2909 2780 2729 2729
query15 202 189 179 179
query16 974 470 478 470
query17 885 732 602 602
query18 2428 431 334 334
query19 216 202 172 172
query20 134 124 125 124
query21 212 133 105 105
query22 13244 13863 14847 13863
query23 16882 16887 16104 16104
query23_1 16083 16018 15701 15701
query24 7145 1630 1226 1226
query24_1 1224 1226 1239 1226
query25 574 476 437 437
query26 1248 269 153 153
query27 2767 480 297 297
query28 4537 1832 1864 1832
query29 871 597 496 496
query30 300 225 187 187
query31 1017 943 867 867
query32 87 73 76 73
query33 538 349 303 303
query34 920 875 525 525
query35 664 702 597 597
query36 1095 1130 993 993
query37 144 98 88 88
query38 2924 2957 2935 2935
query39 872 842 814 814
query39_1 797 815 798 798
query40 240 156 140 140
query41 71 66 66 66
query42 264 269 256 256
query43 247 246 223 223
query44
query45 203 186 185 185
query46 881 987 608 608
query47 2144 2199 2084 2084
query48 311 319 232 232
query49 671 509 387 387
query50 707 274 212 212
query51 4111 4071 3999 3999
query52 260 265 259 259
query53 289 338 280 280
query54 295 270 258 258
query55 89 90 82 82
query56 300 306 310 306
query57 1896 1779 1715 1715
query58 282 268 268 268
query59 2775 2976 2737 2737
query60 327 336 316 316
query61 156 151 157 151
query62 614 586 547 547
query63 307 268 276 268
query64 5095 1276 1020 1020
query65
query66 1452 450 431 431
query67 24266 24316 24208 24208
query68
query69 404 318 288 288
query70 980 926 961 926
query71 335 306 288 288
query72 2846 2628 2400 2400
query73 547 541 314 314
query74 9624 9604 9380 9380
query75 2880 2748 2458 2458
query76 2298 1032 704 704
query77 369 400 302 302
query78 10964 11133 10461 10461
query79 2709 774 581 581
query80 1714 636 551 551
query81 546 267 227 227
query82 989 154 117 117
query83 330 267 246 246
query84 294 121 101 101
query85 904 495 451 451
query86 429 302 328 302
query87 3167 3140 2997 2997
query88 3558 2664 2687 2664
query89 432 368 333 333
query90 2003 183 164 164
query91 164 162 136 136
query92 78 77 71 71
query93 1164 831 495 495
query94 643 328 298 298
query95 594 415 332 332
query96 634 532 230 230
query97 2458 2485 2372 2372
query98 244 226 219 219
query99 1029 991 941 941
Total cold run time: 252473 ms
Total hot run time: 168707 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](azure) Fix incorrect modification timestamps in AzureObjStorage [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #61790:
URL: https://github.com/apache/doris/pull/61790#issuecomment-4139547324
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](azure) Fix incorrect modification timestamps in AzureObjStorage [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring commented on PR #61790:
URL: https://github.com/apache/doris/pull/61790#issuecomment-4139930477
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](azure) Fix incorrect modification timestamps in AzureObjStorage [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring merged PR #61790:
URL: https://github.com/apache/doris/pull/61790
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](azure) Fix incorrect modification timestamps in AzureObjStorage [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #61790:
URL: https://github.com/apache/doris/pull/61790#issuecomment-4140168136
<details>
<summary>TPC-H: <b>Total hot run time: 28215 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ebb7ca45c66b753583a6c74bc31f3e65a143f201, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17616 5268 5104 5104
q2 q3 10644 847 532 532
q4 4678 392 268 268
q5 7577 1250 1018 1018
q6 186 187 157 157
q7 890 924 702 702
q8 9670 1624 1535 1535
q9 5775 5051 5028 5028
q10 6330 1973 1678 1678
q11 482 268 241 241
q12 788 659 502 502
q13 18031 2830 2026 2026
q14 241 248 215 215
q15 q16 754 743 683 683
q17 741 881 432 432
q18 6428 5363 5275 5275
q19 1128 979 618 618
q20 569 505 382 382
q21 4869 2166 1532 1532
q22 397 346 287 287
Total cold run time: 97794 ms
Total hot run time: 28215 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 5813 5631 5688 5631
q2 q3 4081 4447 3930 3930
q4 1091 1460 875 875
q5 4274 4607 4568 4568
q6 222 198 153 153
q7 1916 1665 1541 1541
q8 2896 3098 2966 2966
q9 7912 7772 7605 7605
q10 3964 4186 3689 3689
q11 578 521 440 440
q12 546 644 470 470
q13 2647 3155 2175 2175
q14 296 379 283 283
q15 q16 763 872 722 722
q17 1434 1569 1529 1529
q18 7328 6913 6700 6700
q19 1006 944 949 944
q20 2123 2224 1979 1979
q21 4560 3970 3838 3838
q22 480 449 395 395
Total cold run time: 53930 ms
Total hot run time: 50433 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](azure) Fix incorrect modification timestamps in AzureObjStorage [doris]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #61790:
URL: https://github.com/apache/doris/pull/61790#discussion_r2998597925
##########
fe/fe-core/src/main/java/org/apache/doris/fs/obj/AzureObjStorage.java:
##########
@@ -425,7 +425,8 @@ public Status globList(String remotePath, List<RemoteFile> result, boolean fileN
!isPrefix,
isPrefix ? -1 : blobItem.getProperties().getContentLength(),
isPrefix ? -1 : blobItem.getProperties().getContentLength(),
- isPrefix ? 0 : blobItem.getProperties().getLastModified().getSecond());
+ isPrefix ? 0 : blobItem.getProperties().getLastModified()
+ .toInstant().toEpochMilli());
Review Comment:
Add/extend unit tests (e.g., AzureObjStorageTest’s mocked paged listing) to assert RemoteFile.modificationTime is an epoch-millisecond value derived from BlobItemProperties.getLastModified().toInstant().toEpochMilli(), so regressions back to second-of-minute/epoch-seconds are caught.
##########
fe/fe-core/src/main/java/org/apache/doris/fs/obj/AzureObjStorage.java:
##########
@@ -500,7 +501,7 @@ private Status globListByGetProperties(String bucket, String keyPattern,
props.getBlobSize(),
props.getBlobSize(),
props.getLastModified() != null
- ? props.getLastModified().toEpochSecond() : 0
+ ? props.getLastModified().toInstant().toEpochMilli() : 0
);
Review Comment:
Add test coverage for the deterministic-path getProperties/HEAD flow to validate the modification time unit is epoch milliseconds (not epoch seconds). Using a fixed OffsetDateTime in mocks will make the assertion deterministic.
##########
fe/fe-core/src/main/java/org/apache/doris/fs/obj/AzureObjStorage.java:
##########
@@ -561,7 +562,7 @@ public Status listFiles(String remotePath, boolean recursive, List<RemoteFile> r
false,
props.getContentLength(),
props.getContentLength(),
- props.getLastModified().getSecond(),
+ props.getLastModified().toInstant().toEpochMilli(),
null);
Review Comment:
Consider adding a unit test around listFiles() (or a small isolated helper) to assert BlobItemProperties.getLastModified is converted to epoch milliseconds, since this path previously returned an incorrect value and is easy to regress.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](azure) Fix incorrect modification timestamps in AzureObjStorage [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #61790:
URL: https://github.com/apache/doris/pull/61790#issuecomment-4140219956
# FE UT Coverage Report
Increment line coverage ` 50.00% (2/4)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/61790_ebb7ca45c66b753583a6c74bc31f3e65a143f201/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/61790_ebb7ca45c66b753583a6c74bc31f3e65a143f201/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](azure) Fix incorrect modification timestamps in AzureObjStorage [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #61790:
URL: https://github.com/apache/doris/pull/61790#issuecomment-4140490511
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](azure) Fix incorrect modification timestamps in AzureObjStorage [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #61790:
URL: https://github.com/apache/doris/pull/61790#issuecomment-4140490654
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org