You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "JNSimba (via GitHub)" <gi...@apache.org> on 2026/10/21 03:10:13 UTC
[PR] [fix](fe) Refresh S3 streaming source file metadata [doris]
JNSimba opened a new pull request, #69363:
URL: https://github.com/apache/doris/pull/69363
### What problem does this PR solve?
Issue Number: N/A
Problem Summary:
S3 streaming LIST ingestion drops the modification time when converting listed files to scan metadata. After a failed object is overwritten, a resumed job can reuse cached bytes for the old object because its cache identity has not changed. EVENT ingestion also uses the size from the notification, which may describe an older object and truncate the scan of its replacement.
Preserve the listed modification time and fetch the current object size and modification time with HEAD when accepting an EVENT record. Skip an explicitly missing object; propagate other HEAD failures so uncommitted notifications remain available for retry. Perform HEAD outside the provider lock and discard late batches after close. Objects must still remain unchanged while a batch is being ingested.
Outbound requests / SSRF surface: EVENT processing adds HEAD requests to the S3 endpoint configured in the job's TVF, using the existing object storage client and credentials. Streaming job creation is accessible to users with LOAD privilege on the destination table; SUPER privilege is not required.
### Release note
Fix stale S3 streaming source metadata when retrying ingestion after an object is overwritten.
### Check List (For Author)
- Test:
- [x] Unit Test
- [x] Regression test
- [x] Manual test
- Targeted FE UTs after resolving the rebase: `S3SourceOffsetProviderTest` (7) and `S3EventSourceOffsetProviderTest` (47), 54 tests passed, 0 failures/errors/skipped, using `run-fe-ut.sh --run`.
- Temporary ONE_TIME regression passed five consecutive runs on a shared-nothing 1 FE / 1 BE cluster, with file cache enabled. Each run verified old bad-object cache blocks, replacement object metadata, successful RESUME, all 30 expected rows, and coexistence of old/new cache generations. The temporary suite is not included in this PR.
- FE build and 51 targeted UTs passed before rebasing. After rebasing, the four updated S3 source/test files are tested over the existing developer build (baseline `1a1f93842d8`); the complete reactor with the latest master's dependency versions has not been rebuilt.
- Checkstyle on the rebased PR branch: 0 violations; `git diff --check` passed.
- Shared-storage deployment and EVENT end-to-end ingestion were not tested.
- Behavior changed:
- [x] Yes. EVENT batches use current object metadata rather than notification metadata; missing objects are skipped and other HEAD failures remain retryable.
- Does this need documentation?
- [x] No.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Refresh S3 streaming source file metadata [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69363:
URL: https://github.com/apache/doris/pull/69363#issuecomment-6313596810
<details>
<summary>ClickBench: <b>Total hot run time: 24.28 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit cd09210135e816e4943f063f63b122e6dd0da87a, data reload: false
query1 0.00 0.00 0.00
query2 0.09 0.05 0.05
query3 0.26 0.08 0.08
query4 1.60 0.10 0.10
query5 0.23 0.26 0.24
query6 1.17 0.68 0.67
query7 0.04 0.01 0.00
query8 0.05 0.04 0.04
query9 0.41 0.33 0.36
query10 0.58 0.56 0.55
query11 0.20 0.15 0.14
query12 0.18 0.14 0.15
query13 0.49 0.49 0.49
query14 1.02 1.01 1.00
query15 0.62 0.60 0.60
query16 0.34 0.34 0.34
query17 1.13 1.11 1.09
query18 0.21 0.19 0.19
query19 2.12 1.98 1.95
query20 0.02 0.01 0.01
query21 15.49 0.19 0.13
query22 4.88 0.05 0.05
query23 16.28 0.31 0.12
query24 2.81 0.70 0.41
query25 0.15 0.09 0.06
query26 1.10 0.22 0.16
query27 0.06 0.04 0.07
query28 2.66 0.64 0.35
query29 12.51 4.20 3.29
query30 0.28 0.13 0.10
query31 2.77 0.55 0.33
query32 3.22 0.59 0.50
query33 3.25 3.21 3.20
query34 15.31 4.01 3.39
query35 3.36 3.35 3.33
query36 0.55 0.44 0.44
query37 0.10 0.07 0.06
query38 0.05 0.04 0.04
query39 0.04 0.03 0.03
query40 0.17 0.16 0.15
query41 0.09 0.03 0.03
query42 0.03 0.03 0.03
query43 0.04 0.04 0.03
Total cold run time: 95.96 s
Total hot run time: 24.28 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Refresh S3 streaming source file metadata [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #69363:
URL: https://github.com/apache/doris/pull/69363#discussion_r4332858990
##########
fe/fe-core/src/main/java/org/apache/doris/job/offset/s3/S3EventSourceOffsetProvider.java:
##########
@@ -573,10 +578,22 @@ private ParsedMessage parseMessage(Message message) {
if (isCommitted(key, message.messageId(), message.receiptHandle())) {
continue;
}
+ String path = "s3://" + bucket + "/" + key;
Review Comment:
[P2] Preserve leading slashes in the HEAD object key
For a source `s3://bucket/**`, a notification for the existing key `/a.csv` passes the matcher, but this path becomes `s3://bucket//a.csv` and `S3ObjStorage.headObject` reparses it with `ObjectStorageUri`, which strips the key's leading slash. HEAD therefore checks `a.csv`: if it is absent, the new missing-object catch makes `deleteSkippedMessages` delete a valid notification without ingesting `/a.csv`; if both exist, the scan receives the other object's size/mtime. The previous exact-status BE scan preserves the leading slash. Use a metadata lookup that preserves the exact decoded key and cover `/a.csv` separately from `a.csv`.
##########
fe/fe-core/src/main/java/org/apache/doris/job/offset/s3/S3EventSourceOffsetProvider.java:
##########
@@ -573,10 +578,22 @@ private ParsedMessage parseMessage(Message message) {
if (isCommitted(key, message.messageId(), message.receiptHandle())) {
continue;
}
+ String path = "s3://" + bucket + "/" + key;
+ RemoteObject object;
+ try {
+ // Notifications can outlive an overwrite; refresh both scan size and cache identity.
+ object = objectStorage.headObject(path);
Review Comment:
[P2] Renew receipt visibility during metadata refresh
This HEAD runs serially for every record before `receiveBatch` checks its deadline or publishes a batch that can be renewed. With default `max_interval=10`, the receipt lease is 120 seconds, while a supported receive of two 80-record messages at one second per successful HEAD takes 160 seconds. If INSERT is still running at the next metadata tick, renewal of those expired receipts marks the batch expired and pauses/cancels the task; resume can repeat the same cycle. The file limit also applies only after the whole receive. Keep current and previously accepted receipts renewed while refreshing metadata. The [SQS API](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_ChangeMessageVisibility.html) documents non-inflight/invalid-receipt renewal errors.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Refresh S3 streaming source file metadata [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69363:
URL: https://github.com/apache/doris/pull/69363#issuecomment-6319820131
<details>
<summary>ClickBench: <b>Total hot run time: 24.52 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit bde782aef3aacebbc06dcfff35657bbff1911afa, data reload: false
query1 0.01 0.01 0.00
query2 0.09 0.05 0.05
query3 0.26 0.09 0.09
query4 1.61 0.10 0.10
query5 0.26 0.26 0.25
query6 1.16 0.70 0.69
query7 0.04 0.01 0.01
query8 0.06 0.04 0.04
query9 0.42 0.37 0.38
query10 0.63 0.63 0.64
query11 0.20 0.15 0.14
query12 0.19 0.15 0.15
query13 0.49 0.51 0.51
query14 1.02 1.02 1.02
query15 0.65 0.60 0.64
query16 0.36 0.37 0.36
query17 1.14 1.15 1.16
query18 0.22 0.22 0.25
query19 2.08 2.03 1.88
query20 0.02 0.01 0.01
query21 15.48 0.20 0.14
query22 4.93 0.06 0.05
query23 16.28 0.32 0.12
query24 3.06 0.59 0.41
query25 0.10 0.06 0.05
query26 1.39 0.21 0.14
query27 0.05 0.05 0.04
query28 3.06 0.73 0.38
query29 12.50 4.19 3.31
query30 0.29 0.13 0.12
query31 2.78 0.56 0.32
query32 3.22 0.59 0.49
query33 3.32 3.19 3.29
query34 15.52 4.01 3.39
query35 3.39 3.32 3.36
query36 0.56 0.45 0.43
query37 0.09 0.06 0.07
query38 0.05 0.03 0.04
query39 0.03 0.04 0.03
query40 0.17 0.15 0.14
query41 0.09 0.03 0.03
query42 0.04 0.03 0.03
query43 0.04 0.03 0.03
Total cold run time: 97.35 s
Total hot run time: 24.52 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Refresh S3 streaming source file metadata [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69363:
URL: https://github.com/apache/doris/pull/69363#issuecomment-6328512451
# FE Regression Coverage Report
Increment line coverage ` 52.94% (9/17)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69363_bde782aef3aacebbc06dcfff35657bbff1911afa_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69363_bde782aef3aacebbc06dcfff35657bbff1911afa_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Refresh S3 streaming source file metadata [doris]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba commented on PR #69363:
URL: https://github.com/apache/doris/pull/69363#issuecomment-6310011308
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Refresh S3 streaming source file metadata [doris]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba commented on PR #69363:
URL: https://github.com/apache/doris/pull/69363#issuecomment-6310012590
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Refresh S3 streaming source file metadata [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69363:
URL: https://github.com/apache/doris/pull/69363#issuecomment-6325873058
<details>
<summary>TPC-H: <b>Total hot run time: 28212 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit bde782aef3aacebbc06dcfff35657bbff1911afa, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17607 5131 5032 5032
q2 2097 308 260 260
q3 10246 1367 803 803
q4 4675 461 338 338
q5 7444 793 522 522
q6 177 180 154 154
q7 737 790 597 597
q8 9253 1318 1033 1033
q9 5526 4160 4172 4160
q10 6836 1275 1024 1024
q11 438 235 224 224
q12 640 401 281 281
q13 18095 3563 2976 2976
q14 268 279 240 240
q15 q16 718 718 683 683
q17 1185 737 578 578
q18 6960 6162 6069 6069
q19 1107 949 605 605
q20 375 338 226 226
q21 4824 2148 2243 2148
q22 343 290 259 259
Total cold run time: 99551 ms
Total hot run time: 28212 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 5555 5583 5593 5583
q2 681 568 526 526
q3 5237 5353 4934 4934
q4 1990 2127 1539 1539
q5 5254 5128 5390 5128
q6 254 217 164 164
q7 2412 1991 1843 1843
q8 2996 2695 2649 2649
q9 8085 8049 7876 7876
q10 3724 3659 3267 3267
q11 607 433 580 433
q12 683 719 515 515
q13 3623 3869 3295 3295
q14 320 302 288 288
q15 q16 697 772 670 670
q17 8354 7364 6944 6944
q18 12949 12138 12839 12138
q19 922 813 814 813
q20 2131 2143 1913 1913
q21 5714 4752 4866 4752
q22 501 448 414 414
Total cold run time: 72689 ms
Total hot run time: 65684 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Refresh S3 streaming source file metadata [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69363:
URL: https://github.com/apache/doris/pull/69363#issuecomment-6327425302
# FE Regression Coverage Report
Increment line coverage ` 52.94% (9/17)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69363_bde782aef3aacebbc06dcfff35657bbff1911afa_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69363_bde782aef3aacebbc06dcfff35657bbff1911afa_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Refresh S3 streaming source file metadata [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69363:
URL: https://github.com/apache/doris/pull/69363#issuecomment-6330269815
<details>
<summary>ClickBench: <b>Total hot run time: 24.4 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 95bc35e2bb727b01b08abcba25e3a1ca081ee6f4, data reload: false
query1 0.01 0.01 0.00
query2 0.08 0.05 0.05
query3 0.26 0.08 0.09
query4 1.61 0.10 0.10
query5 0.24 0.24 0.23
query6 1.17 0.68 0.69
query7 0.04 0.00 0.00
query8 0.06 0.04 0.03
query9 0.41 0.36 0.34
query10 0.59 0.57 0.60
query11 0.21 0.15 0.16
query12 0.18 0.14 0.15
query13 0.49 0.49 0.49
query14 1.03 1.02 1.00
query15 0.63 0.60 0.61
query16 0.35 0.34 0.35
query17 1.15 1.17 1.13
query18 0.20 0.20 0.20
query19 2.02 2.02 2.05
query20 0.02 0.01 0.01
query21 15.48 0.21 0.14
query22 4.81 0.06 0.05
query23 16.24 0.32 0.12
query24 2.98 0.59 0.44
query25 0.11 0.06 0.06
query26 1.17 0.22 0.16
query27 0.05 0.03 0.04
query28 3.48 0.77 0.38
query29 12.50 4.08 3.32
query30 0.28 0.13 0.10
query31 2.77 0.58 0.33
query32 3.22 0.59 0.50
query33 3.12 3.17 3.33
query34 15.36 3.96 3.35
query35 3.33 3.32 3.35
query36 0.55 0.46 0.41
query37 0.10 0.06 0.07
query38 0.05 0.04 0.03
query39 0.03 0.03 0.03
query40 0.17 0.14 0.15
query41 0.08 0.04 0.03
query42 0.03 0.04 0.04
query43 0.05 0.03 0.04
Total cold run time: 96.71 s
Total hot run time: 24.4 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Refresh S3 streaming source file metadata [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69363:
URL: https://github.com/apache/doris/pull/69363#issuecomment-6308037258
<details>
<summary>TPC-H: <b>Total hot run time: 28544 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 30f38d9ad3e3956e826cd2e7911f67690e2eedac, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17591 5134 5071 5071
q2 2124 319 259 259
q3 10218 1389 840 840
q4 4679 466 341 341
q5 7437 802 535 535
q6 185 186 161 161
q7 766 802 625 625
q8 9280 1105 1029 1029
q9 5574 4296 4251 4251
q10 6846 1309 1019 1019
q11 435 254 230 230
q12 630 407 285 285
q13 18100 3683 2998 2998
q14 272 274 251 251
q15 q16 736 711 672 672
q17 1194 747 583 583
q18 7078 6183 6102 6102
q19 1257 958 599 599
q20 389 351 226 226
q21 5648 2210 2315 2210
q22 349 297 257 257
Total cold run time: 100788 ms
Total hot run time: 28544 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 5613 5631 5637 5631
q2 679 564 534 534
q3 5213 5368 4931 4931
q4 2014 2159 1576 1576
q5 5233 5160 5729 5160
q6 268 198 153 153
q7 2394 2098 1867 1867
q8 3144 2905 2732 2732
q9 8059 8035 8040 8035
q10 3753 3703 3250 3250
q11 609 437 419 419
q12 668 692 517 517
q13 3516 3955 3279 3279
q14 318 308 299 299
q15 q16 675 712 644 644
q17 8329 7556 7322 7322
q18 12976 12174 12867 12174
q19 942 846 834 834
q20 2133 2150 1929 1929
q21 5659 4621 4728 4621
q22 532 447 457 447
Total cold run time: 72727 ms
Total hot run time: 66354 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Refresh S3 streaming source file metadata [doris]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba commented on PR #69363:
URL: https://github.com/apache/doris/pull/69363#issuecomment-6307244533
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Refresh S3 streaming source file metadata [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69363:
URL: https://github.com/apache/doris/pull/69363#issuecomment-6330075464
<details>
<summary>TPC-H: <b>Total hot run time: 28312 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 95bc35e2bb727b01b08abcba25e3a1ca081ee6f4, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17624 5139 5075 5075
q2 2119 318 250 250
q3 10232 1362 808 808
q4 4679 459 336 336
q5 7435 786 519 519
q6 178 191 154 154
q7 743 803 621 621
q8 9252 1329 1060 1060
q9 5366 4151 4148 4148
q10 6816 1280 1000 1000
q11 443 246 229 229
q12 638 394 274 274
q13 18107 3527 3014 3014
q14 271 273 252 252
q15 q16 729 713 669 669
q17 1165 763 575 575
q18 6922 6160 6093 6093
q19 1104 953 590 590
q20 382 336 228 228
q21 5110 2158 2228 2158
q22 338 294 259 259
Total cold run time: 99653 ms
Total hot run time: 28312 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 5570 5619 5603 5603
q2 663 594 537 537
q3 5248 5383 4958 4958
q4 1950 2136 1510 1510
q5 5289 5126 5439 5126
q6 266 208 155 155
q7 2315 2028 1760 1760
q8 3053 2648 2751 2648
q9 7940 8067 7955 7955
q10 3698 3652 3229 3229
q11 602 432 407 407
q12 721 811 564 564
q13 3458 3846 3246 3246
q14 318 313 294 294
q15 q16 697 703 627 627
q17 8172 7505 6839 6839
q18 12983 12168 12839 12168
q19 883 760 773 760
q20 2133 2136 1922 1922
q21 5590 4561 4735 4561
q22 477 457 407 407
Total cold run time: 72026 ms
Total hot run time: 65276 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Refresh S3 streaming source file metadata [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69363:
URL: https://github.com/apache/doris/pull/69363#issuecomment-6319721891
<details>
<summary>TPC-DS: <b>Total hot run time: 151469 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit bde782aef3aacebbc06dcfff35657bbff1911afa, data reload: false
query5 4336 623 461 461
query6 443 225 212 212
query7 4817 450 236 236
query8 328 179 173 173
query9 8813 3152 3166 3152
query10 492 315 264 264
query11 6005 3892 3604 3604
query12 167 97 93 93
query13 1278 451 341 341
query14 6131 3964 3724 3724
query14_1 3494 3444 3409 3409
query15 209 202 181 181
query16 950 468 450 450
query17 870 675 541 541
query18 2435 425 321 321
query19 207 162 139 139
query20 87 84 88 84
query21 228 140 110 110
query22 12941 13007 12814 12814
query23 13555 12795 12383 12383
query23_1 12607 12507 12557 12507
query24 7378 813 476 476
query24_1 469 481 475 475
query25 550 430 369 369
query26 1261 256 141 141
query27 2768 443 278 278
query28 4612 1927 1926 1926
query29 1586 597 479 479
query30 307 226 186 186
query31 1019 932 827 827
query32 148 96 99 96
query33 527 315 257 257
query34 916 843 493 493
query35 744 766 695 695
query36 798 793 712 712
query37 132 95 87 87
query38 1775 1728 1660 1660
query39 790 803 738 738
query39_1 730 725 740 725
query40 218 113 98 98
query41 66 66 64 64
query42 91 83 81 81
query43 351 365 319 319
query44 1291 680 677 677
query45 182 175 165 165
query46 860 960 572 572
query47 2781 2968 2807 2807
query48 298 303 220 220
query49 574 394 294 294
query50 671 274 218 218
query51 10581 10454 10472 10454
query52 86 86 70 70
query53 188 208 152 152
query54 233 193 177 177
query55 74 71 64 64
query56 249 231 220 220
query57 1670 1600 1546 1546
query58 289 256 263 256
query59 2085 2134 1914 1914
query60 301 247 259 247
query61 149 146 144 144
query62 391 336 303 303
query63 193 160 170 160
query64 2743 941 788 788
query65 3298 3276 3279 3276
query66 1778 419 298 298
query67 20586 20223 20187 20187
query68 3252 984 613 613
query69 386 287 246 246
query70 864 795 749 749
query71 302 240 208 208
query72 2753 2825 2317 2317
query73 538 517 289 289
query74 4663 4487 4362 4362
query75 2357 2294 1902 1902
query76 2375 999 591 591
query77 342 383 290 290
query78 9967 9750 9174 9174
query79 979 837 501 501
query80 1165 439 335 335
query81 598 323 264 264
query82 586 135 110 110
query83 306 195 182 182
query84 309 120 98 98
query85 928 427 418 418
query86 375 239 225 225
query87 2000 1958 1799 1799
query88 3606 2677 2657 2657
query89 325 279 237 237
query90 1856 173 163 163
query91 167 153 128 128
query92 102 97 89 89
query93 987 991 565 565
query94 584 317 277 277
query95 591 323 359 323
query96 658 481 235 235
query97 2429 2478 2318 2318
query98 157 150 148 148
query99 769 754 645 645
Total cold run time: 236291 ms
Total hot run time: 151469 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Refresh S3 streaming source file metadata [doris]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba merged PR #69363:
URL: https://github.com/apache/doris/pull/69363
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Refresh S3 streaming source file metadata [doris]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba commented on PR #69363:
URL: https://github.com/apache/doris/pull/69363#issuecomment-6307263307
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Refresh S3 streaming source file metadata [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69363:
URL: https://github.com/apache/doris/pull/69363#issuecomment-6327821168
# BE Regression && UT Coverage Report
Increment line coverage `100% (0/0)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69363_bde782aef3aacebbc06dcfff35657bbff1911afa_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69363_bde782aef3aacebbc06dcfff35657bbff1911afa_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 77.22% (35878/46460) |
| Line Coverage | 62.47% (404967/648254) |
| Region Coverage | 58.88% (341782/580431) |
| Branch Coverage | 59.90% (158271/264247) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Refresh S3 streaming source file metadata [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69363:
URL: https://github.com/apache/doris/pull/69363#issuecomment-6313509818
<details>
<summary>TPC-DS: <b>Total hot run time: 150530 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit cd09210135e816e4943f063f63b122e6dd0da87a, data reload: false
query5 4336 588 442 442
query6 424 234 195 195
query7 4805 447 217 217
query8 315 163 155 155
query9 8763 3084 3079 3079
query10 479 305 268 268
query11 6036 3868 3576 3576
query12 145 91 89 89
query13 1257 444 320 320
query14 6173 3975 3620 3620
query14_1 3396 3386 3384 3384
query15 207 194 193 193
query16 978 457 424 424
query17 1092 686 550 550
query18 2421 425 331 331
query19 209 172 144 144
query20 111 84 83 83
query21 212 127 105 105
query22 12961 13004 12738 12738
query23 13474 13122 12462 12462
query23_1 12617 12542 12591 12542
query24 7383 811 458 458
query24_1 469 472 470 470
query25 538 428 361 361
query26 1281 272 136 136
query27 2739 438 267 267
query28 4598 1931 1914 1914
query29 1567 580 419 419
query30 298 220 182 182
query31 999 938 801 801
query32 164 86 92 86
query33 515 318 237 237
query34 926 839 492 492
query35 738 747 629 629
query36 792 871 728 728
query37 117 94 78 78
query38 1770 1723 1687 1687
query39 797 798 755 755
query39_1 716 724 736 724
query40 217 113 94 94
query41 67 63 61 61
query42 83 83 79 79
query43 336 359 301 301
query44 1303 672 689 672
query45 185 174 157 157
query46 831 965 566 566
query47 2884 3047 2814 2814
query48 285 298 215 215
query49 574 400 290 290
query50 659 271 203 203
query51 10353 10380 10533 10380
query52 76 77 68 68
query53 187 204 147 147
query54 237 212 184 184
query55 68 67 61 61
query56 234 227 212 212
query57 1653 1628 1572 1572
query58 293 267 255 255
query59 2138 2163 2008 2008
query60 278 240 229 229
query61 145 149 149 149
query62 389 330 284 284
query63 186 155 156 155
query64 2742 927 760 760
query65 3318 3269 3247 3247
query66 1768 429 298 298
query67 20216 19939 20144 19939
query68 3047 986 594 594
query69 388 278 243 243
query70 846 755 749 749
query71 294 243 214 214
query72 2763 2574 2273 2273
query73 515 511 293 293
query74 4613 4511 4378 4378
query75 2354 2284 1911 1911
query76 2322 994 577 577
query77 362 367 279 279
query78 9862 9755 9172 9172
query79 875 903 518 518
query80 547 439 387 387
query81 527 318 277 277
query82 1012 140 107 107
query83 308 204 190 190
query84 275 132 112 112
query85 940 499 418 418
query86 324 238 232 232
query87 1960 1978 1774 1774
query88 3645 2658 2651 2651
query89 325 279 241 241
query90 1846 165 166 165
query91 159 149 126 126
query92 94 90 90 90
query93 993 984 580 580
query94 460 286 272 272
query95 563 390 296 296
query96 654 480 228 228
query97 2463 2427 2312 2312
query98 162 151 147 147
query99 766 760 651 651
Total cold run time: 234535 ms
Total hot run time: 150530 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Refresh S3 streaming source file metadata [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69363:
URL: https://github.com/apache/doris/pull/69363#issuecomment-6327002258
# BE Regression && UT Coverage Report
Increment line coverage `100% (0/0)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69363_bde782aef3aacebbc06dcfff35657bbff1911afa_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69363_bde782aef3aacebbc06dcfff35657bbff1911afa_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 77.22% (35878/46460) |
| Line Coverage | 62.47% (404985/648254) |
| Region Coverage | 58.91% (341929/580431) |
| Branch Coverage | 59.90% (158274/264247) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Refresh S3 streaming source file metadata [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69363:
URL: https://github.com/apache/doris/pull/69363#issuecomment-6328068105
# FE Regression Coverage Report
Increment line coverage ` 52.94% (9/17)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69363_bde782aef3aacebbc06dcfff35657bbff1911afa_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69363_bde782aef3aacebbc06dcfff35657bbff1911afa_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Refresh S3 streaming source file metadata [doris]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba commented on PR #69363:
URL: https://github.com/apache/doris/pull/69363#issuecomment-6313715901
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Refresh S3 streaming source file metadata [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69363:
URL: https://github.com/apache/doris/pull/69363#issuecomment-6313483706
# FE UT Coverage Report
Increment line coverage ` 94.12% (16/17)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69363_cd09210135e816e4943f063f63b122e6dd0da87a/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69363_cd09210135e816e4943f063f63b122e6dd0da87a/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Refresh S3 streaming source file metadata [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69363:
URL: https://github.com/apache/doris/pull/69363#issuecomment-6310128920
# FE Regression Coverage Report
Increment line coverage ` 0.09% (1/1102)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69363_30f38d9ad3e3956e826cd2e7911f67690e2eedac_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69363_30f38d9ad3e3956e826cd2e7911f67690e2eedac_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Refresh S3 streaming source file metadata [doris]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba commented on PR #69363:
URL: https://github.com/apache/doris/pull/69363#issuecomment-6330383066
run external
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Refresh S3 streaming source file metadata [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69363:
URL: https://github.com/apache/doris/pull/69363#issuecomment-6333297935
# BE Regression && UT Coverage Report
Increment line coverage `100% (0/0)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69363_95bc35e2bb727b01b08abcba25e3a1ca081ee6f4_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69363_95bc35e2bb727b01b08abcba25e3a1ca081ee6f4_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 75.19% (34932/46460) |
| Line Coverage | 60.14% (389890/648254) |
| Region Coverage | 56.25% (326475/580431) |
| Branch Coverage | 57.35% (151549/264247) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Refresh S3 streaming source file metadata [doris]
Posted by "liaoxin01 (via GitHub)" <gi...@apache.org>.
liaoxin01 commented on PR #69363:
URL: https://github.com/apache/doris/pull/69363#issuecomment-6359455236
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Refresh S3 streaming source file metadata [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #69363:
URL: https://github.com/apache/doris/pull/69363#discussion_r4343002299
##########
fe/fe-core/src/main/java/org/apache/doris/job/offset/s3/S3EventSourceOffsetProvider.java:
##########
@@ -573,10 +573,10 @@ private ParsedMessage parseMessage(Message message) {
if (isCommitted(key, message.messageId(), message.receiptHandle())) {
continue;
}
+ String path = "s3://" + bucket + "/" + key;
files.add(key);
- // The object must remain unchanged until ingestion completes; notifications identify keys, not versions.
- fileStatuses.add(new TBrokerFileStatus("s3://" + bucket + "/" + key, false,
- record.s3.object.size, true));
+ fileStatuses.add(new TBrokerFileStatus(path, false, record.s3.object.size, true)
+ .setModificationTime(System.currentTimeMillis()));
Review Comment:
[P1] Refresh the byte bound when invalidating an EVENT object's cache
With file caching and an explicit CSV schema, let a 4-byte invalid-UTF-8 object fail ingestion, overwrite it with `1,a\n2,b\n` (8 bytes), then RESUME. If a receive contains the old notification before the replacement notification ([event ordering is not guaranteed](https://docs.aws.amazon.com/AmazonS3/latest/userguide/notification-content-structure.html)), `receiveBatch` keeps the old size 4 for that key but records both receipts. This new mtime bypasses the cached failed bytes; `S3FileReader` trusts the nonnegative `file_size` and clips reads at byte 4, so the task successfully inserts only `1,a\n` and acknowledges both events, losing the second row. Before this change the same warm-cache retry kept failing on the old cached bytes. Refresh the current size together with the cache identity (or reject a changed object), and cover redelivery of the unchanged old notification; the updated resume test instead supplies size 170 in a new notification.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Refresh S3 streaming source file metadata [doris]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba commented on PR #69363:
URL: https://github.com/apache/doris/pull/69363#issuecomment-6328635538
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Refresh S3 streaming source file metadata [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69363:
URL: https://github.com/apache/doris/pull/69363#issuecomment-6326929689
# BE UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/bde782aef3aacebbc06dcfff35657bbff1911afa_bde782aef3aacebbc06dcfff35657bbff1911afa/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/bde782aef3aacebbc06dcfff35657bbff1911afa_bde782aef3aacebbc06dcfff35657bbff1911afa/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 66.46% (31867/47951) |
| Line Coverage | 51.23% (335725/655282) |
| Region Coverage | 46.60% (271475/582551) |
| Branch Coverage | 48.35% (128094/264904) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Refresh S3 streaming source file metadata [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69363:
URL: https://github.com/apache/doris/pull/69363#issuecomment-6327245578
# BE Regression && UT Coverage Report
Increment line coverage `100% (0/0)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69363_bde782aef3aacebbc06dcfff35657bbff1911afa_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69363_bde782aef3aacebbc06dcfff35657bbff1911afa_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 77.22% (35878/46460) |
| Line Coverage | 62.47% (404963/648254) |
| Region Coverage | 58.89% (341788/580431) |
| Branch Coverage | 59.89% (158267/264247) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Refresh S3 streaming source file metadata [doris]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba commented on PR #69363:
URL: https://github.com/apache/doris/pull/69363#issuecomment-6313717435
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Refresh S3 streaming source file metadata [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #69363:
URL: https://github.com/apache/doris/pull/69363#discussion_r4335892976
##########
fe/fe-core/src/main/java/org/apache/doris/tablefunction/ExternalFileTableValuedFunction.java:
##########
@@ -336,17 +336,27 @@ public List<Column> getTableColumns() throws AnalysisException {
TNetworkAddress address = new TNetworkAddress(be.getHost(), be.getBrpcPort());
try {
- PFetchTableSchemaRequest request = getFetchTableStructureRequest();
+ if (getTFileType() == TFileType.FILE_STREAM) {
+ fileStatuses.add(new TBrokerFileStatus("", false, -1, true));
+ }
+ Iterator<TBrokerFileStatus> files = fileStatuses.iterator();
+ PFetchTableSchemaRequest request = getFetchTableStructureRequest(files);
InternalService.PFetchTableSchemaResult result = null;
// `request == null` means we don't need to get schemas from BE,
// and we fill a dummy col for this table.
- if (request != null) {
+ while (request != null) {
Future<InternalService.PFetchTableSchemaResult> future = BackendServiceProxy.getInstance()
.fetchTableStructureAsync(address, request);
result = future.get();
TStatusCode code = TStatusCode.findByValue(result.getStatus().getStatusCode());
+ if (code == TStatusCode.NOT_FOUND) {
Review Comment:
[P2] Update the schema-failure test for NOT_FOUND retries
This retry changes the invocation count assumed by `S3TableValuedFunctionTest.testSchemaFailureDoesNotSkipFiles`. Its NOT_FOUND subcase supplies two nonempty files and returns NOT_FOUND for every schema RPC, so the new loop calls `fetchTableStructureAsync` twice before throwing. `assertSchemaFailure` still uses `Mockito.verify(proxy).fetchTableStructureAsync(...)`, which requires exactly one call, so that existing test now fails. Update its NOT_FOUND expectations alongside the retry behavior, while retaining the single-call assertion for other errors.
##########
fe/fe-core/src/test/java/org/apache/doris/job/offset/s3/S3EventSourceOffsetProviderTest.java:
##########
@@ -271,20 +304,65 @@ public void testSqsCloseFailureStillClosesFileSystem() {
}
@Test
- public void testNotificationMetadataDoesNotRequireHead() {
+ public void testNotificationUsesCurrentObjectMetadata() {
+ Mockito.when(s3Client.headObject(HeadObjectRequest.builder()
+ .bucket("bucket").key("logs/a,b[1].csv").build()))
+ .thenReturn(HeadObjectResponse.builder().contentLength(170L)
+ .lastModified(Instant.parse("2026-01-01T00:00:01Z")).eTag("new-etag").build());
Mockito.when(sqsClient.receiveMessage(Mockito.any(ReceiveMessageRequest.class)))
.thenReturn(response(message("literal", record("bucket", "logs/a%2Cb%5B1%5D.csv", 123))));
provider.fetchRemoteMeta(jobProperties(1), tvfProperties);
S3EventOffset offset = nextOffset();
Assertions.assertEquals(Collections.singletonList("logs/a,b[1].csv"), offset.getFiles());
Assertions.assertEquals("s3://bucket/logs/a,b[1].csv", offset.getFileStatuses().get(0).getPath());
- Assertions.assertEquals(123, offset.getFileStatuses().get(0).getSize());
+ Assertions.assertEquals(170, offset.getFileStatuses().get(0).getSize());
Review Comment:
[P2] Reconcile the HEAD-based tests with the EVENT implementation
These tests cannot pass on this head: `S3EventSourceOffsetProvider.parseMessage` never calls `headObject`, copies `record.s3.object.size`, and assigns `System.currentTimeMillis()`. Here the notification is 123 bytes, so this assertion receives 123 rather than 170. The new missing-object, HEAD-failure, close-during-HEAD, mixed-record and both resume cases likewise depend on unused HEAD stubs and fail their assertions/verifications. Reconcile these tests with the intended implementation before relying on this suite to validate overwrite, error and close behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Refresh S3 streaming source file metadata [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69363:
URL: https://github.com/apache/doris/pull/69363#issuecomment-6326067132
<details>
<summary>TPC-DS: <b>Total hot run time: 150887 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit bde782aef3aacebbc06dcfff35657bbff1911afa, data reload: false
query5 4309 556 456 456
query6 428 201 186 186
query7 4816 444 220 220
query8 317 168 152 152
query9 8762 3068 3035 3035
query10 481 299 254 254
query11 5874 3863 3583 3583
query12 156 86 88 86
query13 1251 466 314 314
query14 6149 3936 3640 3640
query14_1 3411 3419 3441 3419
query15 204 201 197 197
query16 932 450 392 392
query17 862 672 534 534
query18 2420 434 313 313
query19 193 163 143 143
query20 88 85 83 83
query21 212 130 106 106
query22 13014 13053 12769 12769
query23 13435 12794 12380 12380
query23_1 12472 12573 12580 12573
query24 7358 814 463 463
query24_1 474 464 484 464
query25 537 421 356 356
query26 1270 259 141 141
query27 2766 439 273 273
query28 4604 1921 1924 1921
query29 1532 600 452 452
query30 296 218 183 183
query31 1021 926 815 815
query32 134 94 93 93
query33 529 300 241 241
query34 891 850 501 501
query35 737 770 657 657
query36 792 824 740 740
query37 127 102 95 95
query38 1784 1761 1662 1662
query39 785 788 734 734
query39_1 732 731 740 731
query40 219 117 101 101
query41 73 84 68 68
query42 89 85 83 83
query43 355 372 352 352
query44 1323 670 682 670
query45 174 176 155 155
query46 826 959 572 572
query47 2853 2950 2823 2823
query48 289 308 215 215
query49 581 411 291 291
query50 669 271 202 202
query51 11308 11169 10909 10909
query52 86 82 68 68
query53 178 202 147 147
query54 257 190 185 185
query55 71 68 64 64
query56 238 208 223 208
query57 1674 1597 1564 1564
query58 275 261 253 253
query59 2119 2094 1907 1907
query60 284 251 222 222
query61 150 156 134 134
query62 388 332 278 278
query63 190 157 160 157
query64 2725 901 743 743
query65 3303 3251 3244 3244
query66 1793 420 293 293
query67 19946 20024 20076 20024
query68 3041 991 598 598
query69 386 288 242 242
query70 823 771 771 771
query71 296 236 208 208
query72 2733 2460 2293 2293
query73 507 529 297 297
query74 4639 4502 4317 4317
query75 2344 2301 1891 1891
query76 2307 1020 609 609
query77 356 366 271 271
query78 9588 9618 9134 9134
query79 976 843 508 508
query80 1206 413 354 354
query81 570 312 275 275
query82 588 134 104 104
query83 302 197 175 175
query84 306 119 96 96
query85 853 420 409 409
query86 387 237 225 225
query87 2055 1929 1814 1814
query88 3587 2651 2620 2620
query89 333 284 239 239
query90 1914 175 177 175
query91 158 150 123 123
query92 94 86 85 85
query93 1010 1001 571 571
query94 623 290 271 271
query95 588 392 303 303
query96 681 478 233 233
query97 2444 2431 2312 2312
query98 166 149 147 147
query99 777 757 632 632
Total cold run time: 234829 ms
Total hot run time: 150887 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Refresh S3 streaming source file metadata [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #69363:
URL: https://github.com/apache/doris/pull/69363#discussion_r4334147490
##########
fe/fe-core/src/main/java/org/apache/doris/job/offset/s3/S3EventSourceOffsetProvider.java:
##########
@@ -573,10 +578,22 @@ private ParsedMessage parseMessage(Message message) {
if (isCommitted(key, message.messageId(), message.receiptHandle())) {
continue;
}
+ String path = "s3://" + bucket + "/" + key;
+ RemoteObject object;
+ try {
+ // Notifications can outlive an overwrite; refresh both scan size and cache identity.
+ object = objectStorage.headObject(path);
+ } catch (FileNotFoundException e) {
+ log.info("Skip missing S3 EVENT object: jobId={}, messageId={}, path={}",
+ jobId, message.messageId(), path);
+ continue;
+ } catch (IOException e) {
+ throw new IllegalStateException("Failed to read S3 EVENT object metadata: " + path, e);
+ }
files.add(key);
// The object must remain unchanged until ingestion completes; notifications identify keys, not versions.
- fileStatuses.add(new TBrokerFileStatus("s3://" + bucket + "/" + key, false,
- record.s3.object.size, true));
+ fileStatuses.add(new TBrokerFileStatus(path, false, object.getSize(), true)
+ .setModificationTime(object.getModificationTime()));
Review Comment:
[P2] Keep S3 generations distinct in the parsed-footer caches
S3 replacements can share a `Last-Modified` second ([HEAD response](https://docs.aws.amazon.com/AmazonS3/latest/API/API_HeadObject.html), [HTTP-date precision](https://www.rfc-editor.org/rfc/rfc9110.html#section-8.8.2.2)). With block file caching disabled, read a small Parquet A to warm its footer cache, replace it with an equal-length B in that second, then start/resume a streaming job while B stays unchanged. Setting this field enables V2 `ParquetFileContext` caching, which previously bypassed streaming files with mtime=0; its path/mtime/size key now finds A's footer. A predicate matching B but excluded by A's statistics can therefore drop B's rows. With `session.enable_file_scanner_v2=false`, legacy ORC also regresses for different-length replacements: `FileMetaCache::get_key` switches from path+size to path+mtime, discarding the distinguishing size. These regressions occur independently of the preexisting block-cache issue; the LIST assignment has the same consequence when its
timestamps collide. Use an object version identity or bypass these mutable-file caches when generations cannot be distinguished, retain size in legacy keys, and cover same-timestamp replacements.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Refresh S3 streaming source file metadata [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69363:
URL: https://github.com/apache/doris/pull/69363#issuecomment-6330205541
<details>
<summary>TPC-DS: <b>Total hot run time: 150863 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 95bc35e2bb727b01b08abcba25e3a1ca081ee6f4, data reload: false
query5 4301 569 431 431
query6 421 205 189 189
query7 4812 435 246 246
query8 318 162 152 152
query9 8796 3079 3081 3079
query10 484 300 286 286
query11 5954 3869 3560 3560
query12 153 92 90 90
query13 1240 440 308 308
query14 6065 3937 3632 3632
query14_1 3417 3387 3420 3387
query15 202 192 177 177
query16 927 406 398 398
query17 827 621 508 508
query18 2424 414 297 297
query19 184 163 131 131
query20 85 81 80 80
query21 206 127 107 107
query22 12954 13040 12729 12729
query23 13277 12858 12423 12423
query23_1 12512 12429 12426 12426
query24 7395 854 460 460
query24_1 478 472 465 465
query25 537 420 392 392
query26 1254 252 132 132
query27 2771 433 263 263
query28 4632 1902 1904 1902
query29 1531 575 439 439
query30 299 224 175 175
query31 1002 933 814 814
query32 133 85 89 85
query33 522 291 237 237
query34 880 832 490 490
query35 739 756 648 648
query36 809 799 708 708
query37 117 111 77 77
query38 1769 1739 1710 1710
query39 793 814 730 730
query39_1 720 731 744 731
query40 212 118 92 92
query41 64 64 64 64
query42 86 82 81 81
query43 349 375 315 315
query44 1335 672 672 672
query45 172 176 162 162
query46 814 946 559 559
query47 2991 2971 2834 2834
query48 290 309 210 210
query49 582 392 305 305
query50 678 268 219 219
query51 10261 10485 10436 10436
query52 81 80 68 68
query53 183 211 148 148
query54 232 187 194 187
query55 70 67 61 61
query56 233 215 212 212
query57 1668 1637 1562 1562
query58 291 262 263 262
query59 2110 2136 1907 1907
query60 292 254 241 241
query61 169 164 169 164
query62 385 341 286 286
query63 197 161 164 161
query64 2839 1020 866 866
query65 3336 3281 3286 3281
query66 1775 432 310 310
query67 20340 20173 20139 20139
query68 2988 1040 649 649
query69 402 316 259 259
query70 912 791 776 776
query71 299 238 221 221
query72 2882 2562 2305 2305
query73 512 531 305 305
query74 4635 4548 4330 4330
query75 2319 2285 1926 1926
query76 2313 1023 616 616
query77 332 387 271 271
query78 9755 9689 9166 9166
query79 987 844 510 510
query80 1272 440 345 345
query81 584 313 278 278
query82 600 132 105 105
query83 309 195 179 179
query84 309 134 105 105
query85 852 434 381 381
query86 400 238 225 225
query87 2015 1942 1810 1810
query88 3739 2669 2658 2658
query89 334 280 239 239
query90 1862 172 173 172
query91 160 156 132 132
query92 100 89 90 89
query93 1062 1004 581 581
query94 648 291 275 275
query95 583 389 308 308
query96 672 502 234 234
query97 2453 2423 2334 2334
query98 164 154 146 146
query99 755 760 644 644
Total cold run time: 234954 ms
Total hot run time: 150863 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Refresh S3 streaming source file metadata [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69363:
URL: https://github.com/apache/doris/pull/69363#issuecomment-6322953211
# BE Regression && UT Coverage Report
Increment line coverage `100% (0/0)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69363_bde782aef3aacebbc06dcfff35657bbff1911afa_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69363_bde782aef3aacebbc06dcfff35657bbff1911afa_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 77.23% (35879/46460) |
| Line Coverage | 62.48% (404998/648254) |
| Region Coverage | 58.90% (341894/580431) |
| Branch Coverage | 59.89% (158256/264247) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Refresh S3 streaming source file metadata [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69363:
URL: https://github.com/apache/doris/pull/69363#issuecomment-6327439600
# FE Regression Coverage Report
Increment line coverage ` 52.94% (9/17)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69363_bde782aef3aacebbc06dcfff35657bbff1911afa_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69363_bde782aef3aacebbc06dcfff35657bbff1911afa_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Refresh S3 streaming source file metadata [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69363:
URL: https://github.com/apache/doris/pull/69363#issuecomment-6313328622
<details>
<summary>TPC-H: <b>Total hot run time: 28602 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit cd09210135e816e4943f063f63b122e6dd0da87a, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17625 5166 5056 5056
q2 2156 318 274 274
q3 10195 1399 817 817
q4 4688 464 334 334
q5 7442 785 537 537
q6 184 181 149 149
q7 753 816 594 594
q8 9260 1333 1147 1147
q9 5525 4222 4203 4203
q10 6826 1298 1049 1049
q11 438 244 229 229
q12 637 400 280 280
q13 18105 3651 3021 3021
q14 279 271 250 250
q15 q16 744 727 670 670
q17 1197 757 593 593
q18 7047 6176 6103 6103
q19 1123 1007 620 620
q20 384 336 227 227
q21 5314 2205 2279 2205
q22 348 293 244 244
Total cold run time: 100270 ms
Total hot run time: 28602 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 5634 5611 5603 5603
q2 662 558 527 527
q3 5207 5384 4873 4873
q4 2043 2137 1536 1536
q5 5246 5194 5727 5194
q6 271 202 153 153
q7 2387 2083 1898 1898
q8 3060 2829 2664 2664
q9 8055 8028 8060 8028
q10 3783 3692 3229 3229
q11 610 433 411 411
q12 687 685 522 522
q13 3532 4041 3257 3257
q14 301 302 315 302
q15 q16 689 709 641 641
q17 8158 7440 7270 7270
q18 12947 12208 12818 12208
q19 904 804 855 804
q20 2125 2139 1913 1913
q21 5574 4549 4665 4549
q22 502 450 396 396
Total cold run time: 72377 ms
Total hot run time: 65978 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Refresh S3 streaming source file metadata [doris]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba commented on PR #69363:
URL: https://github.com/apache/doris/pull/69363#issuecomment-6313855358
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Refresh S3 streaming source file metadata [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69363:
URL: https://github.com/apache/doris/pull/69363#issuecomment-6333550781
# BE Regression && UT Coverage Report
Increment line coverage `100% (0/0)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69363_95bc35e2bb727b01b08abcba25e3a1ca081ee6f4_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69363_95bc35e2bb727b01b08abcba25e3a1ca081ee6f4_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 75.19% (34932/46460) |
| Line Coverage | 60.14% (389890/648254) |
| Region Coverage | 56.25% (326475/580431) |
| Branch Coverage | 57.35% (151549/264247) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Refresh S3 streaming source file metadata [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69363:
URL: https://github.com/apache/doris/pull/69363#issuecomment-6308218492
<details>
<summary>ClickBench: <b>Total hot run time: 24.26 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 30f38d9ad3e3956e826cd2e7911f67690e2eedac, data reload: false
query1 0.01 0.01 0.01
query2 0.10 0.05 0.05
query3 0.26 0.09 0.09
query4 1.61 0.09 0.13
query5 0.23 0.24 0.24
query6 1.18 0.69 0.67
query7 0.04 0.01 0.01
query8 0.05 0.04 0.04
query9 0.40 0.36 0.34
query10 0.57 0.59 0.58
query11 0.20 0.14 0.14
query12 0.19 0.15 0.15
query13 0.48 0.49 0.50
query14 1.02 1.02 1.03
query15 0.63 0.61 0.60
query16 0.34 0.34 0.35
query17 1.14 1.10 1.10
query18 0.20 0.19 0.18
query19 2.06 1.93 2.00
query20 0.02 0.02 0.01
query21 15.44 0.20 0.13
query22 4.93 0.06 0.05
query23 16.26 0.32 0.12
query24 2.99 0.51 0.38
query25 0.13 0.07 0.06
query26 0.90 0.21 0.16
query27 0.06 0.05 0.04
query28 3.73 0.82 0.37
query29 12.48 4.12 3.24
query30 0.27 0.13 0.10
query31 2.80 0.55 0.33
query32 3.22 0.60 0.51
query33 3.13 3.26 3.17
query34 15.39 4.02 3.40
query35 3.35 3.34 3.36
query36 0.54 0.48 0.43
query37 0.09 0.06 0.06
query38 0.05 0.03 0.04
query39 0.04 0.03 0.03
query40 0.17 0.16 0.14
query41 0.09 0.03 0.03
query42 0.04 0.03 0.03
query43 0.04 0.04 0.03
Total cold run time: 96.87 s
Total hot run time: 24.26 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Refresh S3 streaming source file metadata [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69363:
URL: https://github.com/apache/doris/pull/69363#issuecomment-6326159849
<details>
<summary>ClickBench: <b>Total hot run time: 24.37 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit bde782aef3aacebbc06dcfff35657bbff1911afa, data reload: false
query1 0.00 0.00 0.00
query2 0.09 0.05 0.04
query3 0.26 0.08 0.08
query4 1.62 0.10 0.10
query5 0.23 0.24 0.24
query6 1.16 0.69 0.67
query7 0.03 0.01 0.00
query8 0.05 0.03 0.03
query9 0.39 0.33 0.35
query10 0.57 0.59 0.58
query11 0.20 0.15 0.14
query12 0.18 0.14 0.14
query13 0.49 0.50 0.49
query14 1.02 1.01 1.02
query15 0.62 0.60 0.61
query16 0.34 0.35 0.34
query17 1.13 1.12 1.11
query18 0.20 0.19 0.19
query19 2.11 1.93 1.94
query20 0.02 0.02 0.01
query21 15.48 0.19 0.13
query22 4.92 0.05 0.05
query23 16.26 0.31 0.12
query24 2.94 0.49 0.56
query25 0.09 0.06 0.05
query26 1.38 0.23 0.16
query27 0.04 0.05 0.03
query28 3.67 0.73 0.37
query29 12.50 4.15 3.31
query30 0.28 0.14 0.10
query31 2.76 0.55 0.33
query32 3.22 0.59 0.51
query33 3.16 3.27 3.21
query34 15.32 4.01 3.39
query35 3.33 3.31 3.34
query36 0.56 0.47 0.42
query37 0.09 0.07 0.06
query38 0.04 0.04 0.04
query39 0.04 0.03 0.03
query40 0.17 0.15 0.14
query41 0.08 0.03 0.03
query42 0.04 0.03 0.03
query43 0.05 0.04 0.03
Total cold run time: 97.13 s
Total hot run time: 24.37 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Refresh S3 streaming source file metadata [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69363:
URL: https://github.com/apache/doris/pull/69363#issuecomment-6323181930
# FE Regression Coverage Report
Increment line coverage ` 52.94% (9/17)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69363_bde782aef3aacebbc06dcfff35657bbff1911afa_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69363_bde782aef3aacebbc06dcfff35657bbff1911afa_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Refresh S3 streaming source file metadata [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69363:
URL: https://github.com/apache/doris/pull/69363#issuecomment-6328406757
# BE Regression && UT Coverage Report
Increment line coverage `100% (0/0)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69363_bde782aef3aacebbc06dcfff35657bbff1911afa_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69363_bde782aef3aacebbc06dcfff35657bbff1911afa_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 77.23% (35880/46460) |
| Line Coverage | 62.47% (404991/648254) |
| Region Coverage | 58.88% (341776/580431) |
| Branch Coverage | 59.90% (158274/264247) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Refresh S3 streaming source file metadata [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69363:
URL: https://github.com/apache/doris/pull/69363#issuecomment-6333502488
# FE Regression Coverage Report
Increment line coverage ` 2.55% (9/353)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69363_95bc35e2bb727b01b08abcba25e3a1ca081ee6f4_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69363_95bc35e2bb727b01b08abcba25e3a1ca081ee6f4_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Refresh S3 streaming source file metadata [doris]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba commented on PR #69363:
URL: https://github.com/apache/doris/pull/69363#issuecomment-6330847883
run cloud_p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Refresh S3 streaming source file metadata [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69363:
URL: https://github.com/apache/doris/pull/69363#issuecomment-6333691451
# FE Regression Coverage Report
Increment line coverage ` 2.55% (9/353)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69363_95bc35e2bb727b01b08abcba25e3a1ca081ee6f4_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69363_95bc35e2bb727b01b08abcba25e3a1ca081ee6f4_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Refresh S3 streaming source file metadata [doris]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba commented on PR #69363:
URL: https://github.com/apache/doris/pull/69363#issuecomment-6328636943
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Refresh S3 streaming source file metadata [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69363:
URL: https://github.com/apache/doris/pull/69363#issuecomment-6320975461
# FE UT Coverage Report
Increment line coverage ` 64.71% (11/17)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69363_bde782aef3aacebbc06dcfff35657bbff1911afa/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69363_bde782aef3aacebbc06dcfff35657bbff1911afa/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Refresh S3 streaming source file metadata [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69363:
URL: https://github.com/apache/doris/pull/69363#issuecomment-6319534539
<details>
<summary>TPC-H: <b>Total hot run time: 28875 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit bde782aef3aacebbc06dcfff35657bbff1911afa, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17575 5160 5090 5090
q2 2114 324 281 281
q3 10204 1400 819 819
q4 4682 468 338 338
q5 7446 801 531 531
q6 187 186 155 155
q7 736 800 604 604
q8 9267 1361 1081 1081
q9 5500 4251 4242 4242
q10 6819 1303 1021 1021
q11 429 245 235 235
q12 639 399 285 285
q13 18091 3653 3019 3019
q14 277 276 239 239
q15 q16 730 735 654 654
q17 1203 750 575 575
q18 7038 6169 6122 6122
q19 1108 968 623 623
q20 405 346 221 221
q21 5663 2452 2516 2452
q22 377 325 288 288
Total cold run time: 100490 ms
Total hot run time: 28875 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 6091 5933 6289 5933
q2 732 587 571 571
q3 5516 5760 5284 5284
q4 2058 2194 1587 1587
q5 5854 5889 5759 5759
q6 249 187 143 143
q7 2209 1981 1746 1746
q8 3015 2656 2580 2580
q9 8018 7855 7628 7628
q10 3675 3567 3164 3164
q11 582 422 415 415
q12 680 666 507 507
q13 3330 3620 3046 3046
q14 296 295 259 259
q15 q16 671 712 625 625
q17 7843 7104 6895 6895
q18 12964 12199 12867 12199
q19 935 823 815 815
q20 2154 2130 1929 1929
q21 5710 4603 4871 4603
q22 489 449 406 406
Total cold run time: 73071 ms
Total hot run time: 66094 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Refresh S3 streaming source file metadata [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69363:
URL: https://github.com/apache/doris/pull/69363#issuecomment-6327185038
# BE Regression && UT Coverage Report
Increment line coverage `100% (0/0)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69363_bde782aef3aacebbc06dcfff35657bbff1911afa_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69363_bde782aef3aacebbc06dcfff35657bbff1911afa_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 77.22% (35878/46460) |
| Line Coverage | 62.47% (404985/648254) |
| Region Coverage | 58.91% (341929/580431) |
| Branch Coverage | 59.90% (158274/264247) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Refresh S3 streaming source file metadata [doris]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba commented on PR #69363:
URL: https://github.com/apache/doris/pull/69363#issuecomment-6330849368
run external
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Refresh S3 streaming source file metadata [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69363:
URL: https://github.com/apache/doris/pull/69363#issuecomment-6308159069
<details>
<summary>TPC-DS: <b>Total hot run time: 151839 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 30f38d9ad3e3956e826cd2e7911f67690e2eedac, data reload: false
query5 4317 601 461 461
query6 423 210 192 192
query7 4810 446 232 232
query8 315 165 157 157
query9 8774 3106 3132 3106
query10 480 322 257 257
query11 6027 3887 3579 3579
query12 140 94 90 90
query13 1247 450 320 320
query14 6081 4002 3625 3625
query14_1 3477 3439 3417 3417
query15 203 194 184 184
query16 951 439 413 413
query17 853 634 511 511
query18 2412 419 299 299
query19 201 156 143 143
query20 83 82 83 82
query21 217 126 104 104
query22 13034 13163 12897 12897
query23 13577 12914 12581 12581
query23_1 12668 12641 12611 12611
query24 7347 835 457 457
query24_1 462 471 474 471
query25 509 403 367 367
query26 1268 258 139 139
query27 2772 435 268 268
query28 4631 1904 1921 1904
query29 1559 597 447 447
query30 306 228 182 182
query31 1012 940 827 827
query32 144 96 88 88
query33 530 294 238 238
query34 899 857 501 501
query35 754 759 638 638
query36 800 805 709 709
query37 133 111 84 84
query38 1784 1794 1680 1680
query39 795 814 749 749
query39_1 729 735 742 735
query40 218 118 100 100
query41 88 68 101 68
query42 91 80 80 80
query43 353 365 311 311
query44 1361 676 678 676
query45 179 180 159 159
query46 840 973 591 591
query47 3016 3067 2866 2866
query48 307 304 214 214
query49 570 409 290 290
query50 697 281 210 210
query51 10443 10374 10381 10374
query52 78 86 70 70
query53 190 213 154 154
query54 246 205 184 184
query55 73 66 64 64
query56 253 226 210 210
query57 1651 1621 1560 1560
query58 276 275 249 249
query59 2162 2135 1965 1965
query60 286 246 231 231
query61 159 155 148 148
query62 387 337 286 286
query63 200 166 163 163
query64 2734 976 798 798
query65 3348 3316 3252 3252
query66 1809 438 317 317
query67 20277 20140 20067 20067
query68 3273 1028 605 605
query69 410 312 260 260
query70 878 804 772 772
query71 309 247 228 228
query72 2928 2737 2455 2455
query73 536 538 304 304
query74 4641 4547 4381 4381
query75 2351 2308 1948 1948
query76 2402 1033 667 667
query77 344 376 278 278
query78 10058 9788 9193 9193
query79 988 870 525 525
query80 1198 437 367 367
query81 583 319 270 270
query82 574 138 112 112
query83 307 211 207 207
query84 308 125 104 104
query85 874 433 377 377
query86 397 239 232 232
query87 2002 1957 1832 1832
query88 3665 2694 2669 2669
query89 328 286 264 264
query90 1905 175 175 175
query91 167 164 128 128
query92 100 86 83 83
query93 1022 1024 574 574
query94 628 286 277 277
query95 576 327 369 327
query96 669 496 233 233
query97 2453 2453 2302 2302
query98 165 152 148 148
query99 769 766 636 636
Total cold run time: 236680 ms
Total hot run time: 151839 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Refresh S3 streaming source file metadata [doris]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba commented on PR #69363:
URL: https://github.com/apache/doris/pull/69363#issuecomment-6313856695
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Refresh S3 streaming source file metadata [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69363:
URL: https://github.com/apache/doris/pull/69363#issuecomment-6307064604
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://doris.apache.org/community/how-to-contribute/review-and-ci).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Refresh S3 streaming source file metadata [doris]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #69363:
URL: https://github.com/apache/doris/pull/69363#discussion_r4332943979
##########
fe/fe-core/src/main/java/org/apache/doris/job/offset/s3/S3EventSourceOffsetProvider.java:
##########
@@ -573,10 +578,22 @@ private ParsedMessage parseMessage(Message message) {
if (isCommitted(key, message.messageId(), message.receiptHandle())) {
continue;
}
+ String path = "s3://" + bucket + "/" + key;
+ RemoteObject object;
+ try {
+ // Notifications can outlive an overwrite; refresh both scan size and cache identity.
+ object = objectStorage.headObject(path);
Review Comment:
Building an S3 URI via string concatenation can produce invalid/ambiguous URIs for keys containing characters like spaces, `?`, or `#` (URI reserved characters). Since this class already imports `ObjectStorageUri`, consider constructing the location via the repository’s URI helper (e.g., `ObjectStorageUri`) and using its normalized string form for both `headObject(...)` and `TBrokerFileStatus` to avoid subtle parsing/encoding bugs.
##########
fe/fe-core/src/test/java/org/apache/doris/job/offset/s3/S3SourceOffsetProviderTest.java:
##########
@@ -125,6 +125,31 @@ public void testOnceMarksLastBatchAndRecovers() throws Exception {
}
}
+ @Test
+ public void testListingRefreshesFileSizeAndModificationTime() throws Exception {
+ FileSystem fs = Mockito.mock(FileSystem.class);
+ try (MockedStatic<FileSystemFactory> factory = Mockito.mockStatic(FileSystemFactory.class)) {
+ factory.when(() -> FileSystemFactory.getFileSystem(Mockito.any(StorageAdapter.class))).thenReturn(fs);
+ for (S3SourceOffsetProvider provider : List.of(
+ new S3SourceOffsetProvider(), new S3SourceOffsetProvider(ONCE_PROPS))) {
+ Mockito.when(fs.globListWithLimit(Mockito.any(Location.class), Mockito.isNull(),
+ Mockito.anyLong(), Mockito.anyLong())).thenReturn(
+ new GlobListing(List.of(new FileEntry(Location.of("s3://bucket/data/a.csv"),
+ 19, false, 1767225600000L, null)), "bucket", "data/", "data/a.csv"),
+ new GlobListing(List.of(new FileEntry(Location.of("s3://bucket/data/a.csv"),
+ 170, false, 1767225601000L, null)), "bucket", "data/", "data/a.csv"));
Review Comment:
The epoch-millis literals make the test harder to read/verify. Consider deriving these values from `Instant.parse(...)` (or introducing a named constant like `Instant.parse(\"2026-01-01T00:00:00Z\").toEpochMilli()`) so it’s clear what timestamps the test is asserting.
##########
fe/fe-core/src/test/java/org/apache/doris/job/offset/s3/S3SourceOffsetProviderTest.java:
##########
@@ -125,6 +125,31 @@ public void testOnceMarksLastBatchAndRecovers() throws Exception {
}
}
+ @Test
+ public void testListingRefreshesFileSizeAndModificationTime() throws Exception {
+ FileSystem fs = Mockito.mock(FileSystem.class);
+ try (MockedStatic<FileSystemFactory> factory = Mockito.mockStatic(FileSystemFactory.class)) {
+ factory.when(() -> FileSystemFactory.getFileSystem(Mockito.any(StorageAdapter.class))).thenReturn(fs);
+ for (S3SourceOffsetProvider provider : List.of(
+ new S3SourceOffsetProvider(), new S3SourceOffsetProvider(ONCE_PROPS))) {
+ Mockito.when(fs.globListWithLimit(Mockito.any(Location.class), Mockito.isNull(),
+ Mockito.anyLong(), Mockito.anyLong())).thenReturn(
+ new GlobListing(List.of(new FileEntry(Location.of("s3://bucket/data/a.csv"),
+ 19, false, 1767225600000L, null)), "bucket", "data/", "data/a.csv"),
+ new GlobListing(List.of(new FileEntry(Location.of("s3://bucket/data/a.csv"),
+ 170, false, 1767225601000L, null)), "bucket", "data/", "data/a.csv"));
+
+ TBrokerFileStatus first = provider.getNextOffset(ONCE_PROPS, TVF_PROPS).getFileStatuses().get(0);
+ Assertions.assertEquals(19, first.getSize());
+ Assertions.assertEquals(1767225600000L, first.getModificationTime());
Review Comment:
The epoch-millis literals make the test harder to read/verify. Consider deriving these values from `Instant.parse(...)` (or introducing a named constant like `Instant.parse(\"2026-01-01T00:00:00Z\").toEpochMilli()`) so it’s clear what timestamps the test is asserting.
##########
fe/fe-core/src/test/java/org/apache/doris/job/offset/s3/S3SourceOffsetProviderTest.java:
##########
@@ -125,6 +125,31 @@ public void testOnceMarksLastBatchAndRecovers() throws Exception {
}
}
+ @Test
+ public void testListingRefreshesFileSizeAndModificationTime() throws Exception {
+ FileSystem fs = Mockito.mock(FileSystem.class);
+ try (MockedStatic<FileSystemFactory> factory = Mockito.mockStatic(FileSystemFactory.class)) {
+ factory.when(() -> FileSystemFactory.getFileSystem(Mockito.any(StorageAdapter.class))).thenReturn(fs);
+ for (S3SourceOffsetProvider provider : List.of(
+ new S3SourceOffsetProvider(), new S3SourceOffsetProvider(ONCE_PROPS))) {
+ Mockito.when(fs.globListWithLimit(Mockito.any(Location.class), Mockito.isNull(),
+ Mockito.anyLong(), Mockito.anyLong())).thenReturn(
+ new GlobListing(List.of(new FileEntry(Location.of("s3://bucket/data/a.csv"),
+ 19, false, 1767225600000L, null)), "bucket", "data/", "data/a.csv"),
+ new GlobListing(List.of(new FileEntry(Location.of("s3://bucket/data/a.csv"),
+ 170, false, 1767225601000L, null)), "bucket", "data/", "data/a.csv"));
+
+ TBrokerFileStatus first = provider.getNextOffset(ONCE_PROPS, TVF_PROPS).getFileStatuses().get(0);
+ Assertions.assertEquals(19, first.getSize());
+ Assertions.assertEquals(1767225600000L, first.getModificationTime());
+ // An uncommitted batch must pick up an overwrite on the next attempt.
+ TBrokerFileStatus retried = provider.getNextOffset(ONCE_PROPS, TVF_PROPS).getFileStatuses().get(0);
+ Assertions.assertEquals(170, retried.getSize());
+ Assertions.assertEquals(1767225601000L, retried.getModificationTime());
Review Comment:
The epoch-millis literals make the test harder to read/verify. Consider deriving these values from `Instant.parse(...)` (or introducing a named constant like `Instant.parse(\"2026-01-01T00:00:00Z\").toEpochMilli()`) so it’s clear what timestamps the test is asserting.
##########
fe/fe-core/src/main/java/org/apache/doris/job/offset/s3/S3EventSourceOffsetProvider.java:
##########
@@ -573,10 +578,22 @@ private ParsedMessage parseMessage(Message message) {
if (isCommitted(key, message.messageId(), message.receiptHandle())) {
continue;
}
+ String path = "s3://" + bucket + "/" + key;
+ RemoteObject object;
+ try {
+ // Notifications can outlive an overwrite; refresh both scan size and cache identity.
+ object = objectStorage.headObject(path);
+ } catch (FileNotFoundException e) {
+ log.info("Skip missing S3 EVENT object: jobId={}, messageId={}, path={}",
+ jobId, message.messageId(), path);
+ continue;
Review Comment:
Logging missing objects at INFO may be noisy in environments where S3 EVENT messages can be delayed and objects may be short-lived or overwritten frequently. Consider reducing this to DEBUG (or adding throttling / structured counters) so production logs aren't flooded while still keeping the skip behavior observable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Refresh S3 streaming source file metadata [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #69363:
URL: https://github.com/apache/doris/pull/69363#discussion_r4357160361
##########
fe/fe-core/src/main/java/org/apache/doris/job/offset/s3/S3SourceOffsetProvider.java:
##########
@@ -91,7 +91,8 @@ public S3Offset getNextOffset(StreamingJobProperties jobProps, Map<String, Strin
String beginFile = rfiles.get(0).location().uri().replace(bucketBase, "");
String lastFile = rfiles.get(rfiles.size() - 1).location().uri().replace(bucketBase, "");
offset.setFileStatuses(rfiles.stream().map(entry -> new TBrokerFileStatus(
- entry.location().uri(), entry.isDirectory(), entry.length(), !entry.isDirectory()))
+ entry.location().uri(), entry.isDirectory(), entry.length(), !entry.isDirectory())
+ .setModificationTime(entry.modificationTime()))
Review Comment:
[P2] Scope native cache identities to the configured S3 endpoint
The newly nonzero mtime enables V2 Parquet's native footer/page caches for streaming LIST files, but their key uses only fs_name, path, mtime and size. For FILE_S3, fs_name is empty and S3FileReader's path is just s3://bucket/key; FileFactory adds the configured endpoint only to the lower block-cache key. On the same BE, two S3-compatible endpoints containing different objects at the same bucket/key with equal listed mtime/size therefore share native metadata. If A warms a footer with id min/max 1 and B contains id=2 with the same schema, B's WHERE id=2 can prune its matching row group even while both objects remain unchanged. Before this assignment, streaming's zero mtime bypassed these mutable-file native caches. This differs from the existing same-source overwrite collision: include the endpoint/storage namespace in the native identity, or keep these caches bypassed until it is available, and cover two endpoints with matching object metadata.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Refresh S3 streaming source file metadata [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #69363:
URL: https://github.com/apache/doris/pull/69363#issuecomment-6361926027
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org