You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "Gabriel39 (via GitHub)" <gi...@apache.org> on 2026/10/12 05:12:47 UTC
[PR] [feat](datalake) Support nanosecond timestamps in Iceberg and Paimon [doris]
Gabriel39 opened a new pull request, #68934:
URL: https://github.com/apache/doris/pull/68934
### What problem does this PR solve?
Iceberg `timestamp_ns` was unsupported, and Paimon `timestamp(9)` was exposed as `DATETIMEV2(6)`, losing the final three fractional digits. Map these unzoned types bidirectionally to Doris `TIMESTAMP_NS` and preserve all nine digits through reads, inserts and CTAS, including arrays, maps and structs.
- Add nanosecond materialization to Parquet/ORC readers and Iceberg writers. Nanosecond Iceberg schemas use Parquet 2.6; ORC uses a UTC wall-clock carrier to avoid historical timezone-offset differences at the signed nanosecond boundaries.
- Support Iceberg identity, calendar and bucket partitions, including partition-value commit serialization. Omit unsafe ORC timestamp bounds and derived MTMV RANGE boundaries; Iceberg's own partition pruning remains available.
- Select Paimon's JNI reader for projected or filter-only high-precision fields, including nested fields, and use nanosecond Arrow transport for writes. Paimon precisions 7 and 8 also map to `TIMESTAMP_NS`.
Iceberg nanosecond schemas require format version 3. High-precision Paimon reads use JNI, which may cost more than native scans. The existing ORC-645 rejection of ambiguous pre-epoch fractions is retained; Parquet supports those values.
### Release note
Support reading and writing Iceberg `timestamp_ns` and Paimon `timestamp(9)` as Doris `TIMESTAMP_NS`, preserving nanosecond precision.
### Validation
- Full FE and ASAN BE builds; FE Checkstyle, clang-format 16 and clang-tidy over all 29 changed C++ files (initial findings fixed and rechecked).
- 2,112 relevant Java tests: zero failures, five existing skips.
- 117 relevant BE tests under ASAN, including independent Arrow/ORC file read-back, signed bounds, dictionary decoding and overflow handling.
- Three end-to-end regression suites passed against the generated goldens: `test_iceberg_timestamp_ns`, `test_paimon_timestamp_ns`, and `paimon_timestamp_types`. Coverage includes both scanners, session time zones, nested values, filters, signed bounds, CTAS, Iceberg partition transforms/overwrite, and Paimon primary-key upsert.
- Independent Flink read-back of four Doris-written Paimon tables: 28 rows across Parquet/ORC and append/primary-key tables, with exact nanosecond bounds and upserts.
### Check List (For Author)
- Test
- [x] Regression test
- [x] Unit Test
- [x] Manual test (independent Flink read-back described above)
- Behavior changed:
- [x] Yes. External unzoned nanosecond timestamps retain nine fractional digits.
- Does this need documentation?
- [x] Yes. The external type-mapping tables should describe `TIMESTAMP_NS`.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](datalake) Support nanosecond timestamps in Iceberg and Paimon [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #68934:
URL: https://github.com/apache/doris/pull/68934#discussion_r4254819836
##########
be/src/format_v2/parquet/native_schema_desc.cpp:
##########
@@ -882,6 +882,11 @@ std::pair<DataTypePtr, bool> NativeFieldDescriptor::convert_to_doris_type(
// sub-second values are silently truncated by the target SerDe.
ans.first = DataTypeFactory::instance().create_data_type(TYPE_TIMEV2, nullable, 0, scale);
} else if (logicalType.__isset.TIMESTAMP) {
+ if (logicalType.TIMESTAMP.unit.__isset.NANOS && !logicalType.TIMESTAMP.isAdjustedToUTC) {
+ // Materialize nanoseconds before any table cast; DATETIMEV2 would discard the remainder.
+ ans.first = DataTypeFactory::instance().create_data_type(TYPE_TIMESTAMP_NS, nullable);
Review Comment:
[P2] Preserve fused nullable selection for nanosecond columns. This switches optional unadjusted NANOS fields from ColumnDateTimeV2 to the distinct ColumnTimeStampNs, but visit_nullable_expandable_column in format_v2/parquet/reader/native/column_chunk_reader.cpp does not include that class. An integer-filtered scan projecting ts with alternating NULL/value rows therefore loses supports_fused_nullable_selection and falls into decode_selected_values<true>, invoking the decoder and SerDe once per non-NULL run instead of once per page fragment. This regresses the existing sparse nullable scalar path for large files. Register ColumnTimeStampNs with the existing in-place expansion dispatcher and cover a filtered nullable NANOS page with alternating NULLs, checking that HybridSelectionNullFallbackBatches stays zero.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](datalake) Support nanosecond timestamps in Iceberg and Paimon [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68934:
URL: https://github.com/apache/doris/pull/68934#issuecomment-6144517029
<details>
<summary>TPC-H: <b>Total hot run time: 29098 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 19ef150201f4c4928c97b1efbe6003db6b83caf3, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17617 5826 5559 5559
q2 2100 343 254 254
q3 10228 1418 858 858
q4 4696 477 345 345
q5 7429 805 540 540
q6 190 193 156 156
q7 762 798 602 602
q8 9259 1405 1014 1014
q9 5773 4461 4454 4454
q10 6827 1299 1040 1040
q11 432 253 241 241
q12 623 399 288 288
q13 18062 3060 2395 2395
q14 278 271 248 248
q15 q16 722 740 705 705
q17 1229 742 576 576
q18 7190 6249 6211 6211
q19 1107 966 594 594
q20 386 334 232 232
q21 5729 2482 2572 2482
q22 401 353 304 304
Total cold run time: 101040 ms
Total hot run time: 29098 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 6652 6674 6803 6674
q2 745 620 569 569
q3 5597 5803 5413 5413
q4 2105 2142 1567 1567
q5 5740 5605 5734 5605
q6 250 186 143 143
q7 2275 2031 1775 1775
q8 3135 2905 2692 2692
q9 8315 8188 8040 8040
q10 3661 3615 3264 3264
q11 589 398 376 376
q12 665 689 502 502
q13 2741 3094 2401 2401
q14 291 288 278 278
q15 q16 657 676 611 611
q17 7777 6993 6866 6866
q18 13090 12253 13128 12253
q19 920 790 791 790
q20 2194 2184 1930 1930
q21 5822 4704 4774 4704
q22 503 466 398 398
Total cold run time: 73724 ms
Total hot run time: 66851 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](datalake) Support nanosecond timestamps in Iceberg and Paimon [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68934:
URL: https://github.com/apache/doris/pull/68934#issuecomment-6144684844
<details>
<summary>TPC-DS: <b>Total hot run time: 152536 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 19ef150201f4c4928c97b1efbe6003db6b83caf3, data reload: false
query5 4303 611 460 460
query6 428 210 197 197
query7 4805 441 224 224
query8 315 182 163 163
query9 8757 3955 3945 3945
query10 474 305 274 274
query11 5985 3436 3219 3219
query12 144 85 83 83
query13 1254 443 322 322
query14 6525 4766 4486 4486
query14_1 4238 4267 4239 4239
query15 210 205 196 196
query16 983 481 429 429
query17 846 677 561 561
query18 2427 438 310 310
query19 198 166 135 135
query20 85 82 81 81
query21 216 133 111 111
query22 12924 13139 12802 12802
query23 13248 12551 12124 12124
query23_1 12372 12221 12305 12221
query24 7088 833 465 465
query24_1 470 479 488 479
query25 532 423 360 360
query26 1251 258 142 142
query27 2762 440 273 273
query28 4611 1946 1941 1941
query29 1547 594 450 450
query30 294 224 188 188
query31 897 764 659 659
query32 147 97 94 94
query33 505 311 253 253
query34 910 859 518 518
query35 753 768 675 675
query36 834 809 761 761
query37 136 105 94 94
query38 1794 1777 1706 1706
query39 708 727 695 695
query39_1 648 648 684 648
query40 219 120 104 104
query41 71 70 68 68
query42 87 85 84 84
query43 373 374 333 333
query44 1294 705 710 705
query45 181 176 168 168
query46 863 957 578 578
query47 2957 3002 2956 2956
query48 318 310 258 258
query49 558 383 299 299
query50 704 282 212 212
query51 10326 10524 10449 10449
query52 78 79 69 69
query53 192 208 154 154
query54 226 193 172 172
query55 77 69 65 65
query56 217 215 218 215
query57 1586 1594 1576 1576
query58 277 262 252 252
query59 2328 2373 2140 2140
query60 276 236 228 228
query61 151 150 150 150
query62 397 339 283 283
query63 193 169 165 165
query64 2730 951 771 771
query65 3441 3373 3372 3372
query66 1786 403 309 309
query67 20182 20025 20180 20025
query68 3187 995 619 619
query69 379 281 235 235
query70 893 871 824 824
query71 287 223 205 205
query72 2803 2598 2298 2298
query73 509 541 295 295
query74 4556 4482 4305 4305
query75 2376 2285 1949 1949
query76 2295 1017 623 623
query77 354 393 307 307
query78 9013 9116 8547 8547
query79 945 887 514 514
query80 573 422 357 357
query81 551 317 272 272
query82 568 131 108 108
query83 287 192 174 174
query84 307 122 100 100
query85 799 435 362 362
query86 320 234 230 230
query87 2004 1971 1855 1855
query88 3593 2707 2665 2665
query89 336 309 258 258
query90 1943 178 179 178
query91 157 145 127 127
query92 86 84 80 80
query93 992 1012 579 579
query94 446 311 286 286
query95 599 407 313 313
query96 672 516 232 232
query97 2489 2465 2302 2302
query98 155 145 149 145
query99 779 764 661 661
Total cold run time: 233585 ms
Total hot run time: 152536 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](datalake) Support nanosecond timestamps in Iceberg and Paimon [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68934:
URL: https://github.com/apache/doris/pull/68934#issuecomment-6146015056
<details>
<summary>TPC-DS: <b>Total hot run time: 152790 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 19ef150201f4c4928c97b1efbe6003db6b83caf3, data reload: false
query5 4298 577 459 459
query6 421 211 196 196
query7 4811 446 238 238
query8 326 183 174 174
query9 8760 4054 4056 4054
query10 458 320 265 265
query11 5813 3443 3231 3231
query12 147 89 84 84
query13 1252 450 340 340
query14 6546 4818 4523 4523
query14_1 4298 4309 4289 4289
query15 207 200 184 184
query16 940 458 424 424
query17 872 671 546 546
query18 2432 421 317 317
query19 195 168 133 133
query20 82 86 81 81
query21 215 151 120 120
query22 13024 13338 12836 12836
query23 13181 12633 12102 12102
query23_1 12203 12243 12219 12219
query24 7117 859 467 467
query24_1 492 472 480 472
query25 542 421 370 370
query26 1283 258 145 145
query27 2753 444 272 272
query28 4610 1950 1945 1945
query29 1556 603 464 464
query30 297 217 187 187
query31 908 770 656 656
query32 135 99 94 94
query33 528 314 250 250
query34 905 844 508 508
query35 755 769 654 654
query36 818 795 732 732
query37 144 107 98 98
query38 1820 1751 1727 1727
query39 700 750 677 677
query39_1 661 697 707 697
query40 227 122 102 102
query41 73 70 70 70
query42 88 86 86 86
query43 373 383 339 339
query44 1348 715 714 714
query45 190 181 164 164
query46 870 980 578 578
query47 2931 2999 2916 2916
query48 302 321 220 220
query49 576 431 292 292
query50 681 270 214 214
query51 11034 10523 10376 10376
query52 79 83 72 72
query53 193 206 160 160
query54 231 203 173 173
query55 70 66 65 65
query56 244 233 208 208
query57 1618 1597 1576 1576
query58 288 256 249 249
query59 2367 2397 2155 2155
query60 282 238 219 219
query61 143 151 146 146
query62 391 346 302 302
query63 204 165 169 165
query64 2782 992 832 832
query65 3484 3413 3426 3413
query66 1778 413 300 300
query67 20237 20013 20067 20013
query68 3134 1010 639 639
query69 380 284 242 242
query70 916 828 839 828
query71 299 240 210 210
query72 2714 2548 2251 2251
query73 540 545 300 300
query74 4605 4489 4326 4326
query75 2384 2317 1970 1970
query76 2321 1042 654 654
query77 360 392 305 305
query78 9030 9117 8484 8484
query79 952 909 525 525
query80 546 438 360 360
query81 541 331 276 276
query82 587 142 109 109
query83 279 214 193 193
query84 317 123 105 105
query85 847 421 372 372
query86 323 235 228 228
query87 2032 1949 1826 1826
query88 3624 2717 2663 2663
query89 342 300 260 260
query90 1858 184 170 170
query91 160 154 121 121
query92 97 88 84 84
query93 1047 988 581 581
query94 456 312 294 294
query95 590 349 381 349
query96 673 521 235 235
query97 2453 2433 2337 2337
query98 157 148 145 145
query99 779 760 647 647
Total cold run time: 234671 ms
Total hot run time: 152790 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](datalake) Support nanosecond timestamps in Iceberg and Paimon [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #68934:
URL: https://github.com/apache/doris/pull/68934#issuecomment-6118883221
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](datalake) Support nanosecond timestamps in Iceberg and Paimon [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68934:
URL: https://github.com/apache/doris/pull/68934#issuecomment-6136850950
<details>
<summary>ClickBench: <b>Total hot run time: 24.6 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 203abf3a6b955d390f970ace3094d1ddce7bc37a, data reload: false
query1 0.01 0.01 0.00
query2 0.10 0.05 0.05
query3 0.26 0.12 0.12
query4 1.62 0.12 0.12
query5 0.28 0.24 0.25
query6 1.18 0.68 0.68
query7 0.04 0.01 0.01
query8 0.05 0.04 0.04
query9 0.42 0.35 0.35
query10 0.62 0.57 0.59
query11 0.21 0.14 0.13
query12 0.19 0.15 0.14
query13 0.49 0.49 0.50
query14 0.97 0.97 0.96
query15 0.64 0.60 0.61
query16 0.35 0.32 0.35
query17 1.12 1.10 1.15
query18 0.23 0.21 0.21
query19 2.11 2.04 2.04
query20 0.02 0.02 0.02
query21 15.50 0.25 0.14
query22 4.44 0.06 0.05
query23 16.26 0.33 0.12
query24 2.77 0.80 0.45
query25 0.12 0.06 0.04
query26 1.89 0.25 0.14
query27 0.05 0.03 0.04
query28 2.05 0.63 0.41
query29 12.52 4.18 3.34
query30 0.28 0.14 0.16
query31 2.77 0.58 0.33
query32 3.22 0.61 0.52
query33 3.23 3.24 3.19
query34 15.47 4.04 3.38
query35 3.38 3.34 3.34
query36 0.56 0.44 0.48
query37 0.09 0.06 0.07
query38 0.05 0.04 0.04
query39 0.04 0.03 0.03
query40 0.17 0.15 0.14
query41 0.08 0.03 0.02
query42 0.04 0.03 0.03
query43 0.05 0.03 0.03
Total cold run time: 95.94 s
Total hot run time: 24.6 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](datalake) Support nanosecond timestamps in Iceberg and Paimon [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #68934:
URL: https://github.com/apache/doris/pull/68934#discussion_r4254681379
##########
be/src/format_v2/orc/orc_reader.cpp:
##########
@@ -1646,24 +1715,91 @@ Status OrcReader::_select_stripe_ranges_by_statistics() {
return Status::OK();
}
-void OrcReader::_apply_current_stripe_range() {
- if (!_state->stripe_pruning_applied || _state->selected_stripe_ranges.empty()) {
- return;
+Status OrcReader::_prepare_timestamp_stripe_ranges() {
+ if (!_state->match_writer_timezone) {
+ return Status::OK();
+ }
+ std::vector<uint64_t> stripes;
+ if (_state->stripe_ranges_applied) {
+ for (const auto& range : _state->selected_stripe_ranges) {
+ for (auto stripe = range.first_stripe; stripe < range.last_stripe; ++stripe) {
+ stripes.push_back(stripe);
+ }
+ }
+ } else {
+ RETURN_IF_ERROR(_collect_split_stripes(&stripes));
+ }
+ _state->selected_stripe_ranges.clear();
+ _state->stripe_ranges_applied = true;
+ _reader_statistics.read_row_groups = cast_set<int32_t>(stripes.size());
+ if (stripes.empty()) {
+ return Status::OK();
+ }
+ try {
+ uint64_t first_row = 0;
+ size_t selected = 0;
+ for (uint64_t index = 0; index <= stripes.back(); ++index) {
+ const auto stripe = _state->reader->getStripe(index);
+ const auto end_row = first_row + stripe->getNumberOfRows();
+ if (index == stripes[selected]) {
+ // A RowReader has one timezone, whereas writer timezones belong to stripes.
+ // Read footer timezone metadata only when entering the selected stripe.
+ _state->selected_stripe_ranges.push_back({
+ .first_stripe = index,
+ .last_stripe = index + 1,
+ .offset = stripe->getOffset(),
+ .length = stripe->getLength(),
+ .first_row = first_row,
+ .end_row = end_row,
+ });
+ ++selected;
+ }
+ first_row = end_row;
+ }
+ } catch (const std::exception& e) {
+ return Status::InternalError("Failed to prepare ORC timestamp stripe ranges: {}", e.what());
+ }
+ return Status::OK();
+}
+
+Status OrcReader::_apply_current_stripe_range() {
+ if (!_state->stripe_ranges_applied || _state->selected_stripe_ranges.empty()) {
+ return Status::OK();
}
DORIS_CHECK(_state->current_stripe_range < _state->selected_stripe_ranges.size());
const auto& stripe_range = _state->selected_stripe_ranges[_state->current_stripe_range];
_state->row_reader_options.range(stripe_range.offset, stripe_range.length);
+ if (_state->match_writer_timezone) {
+ try {
+ const auto stripe = _state->reader->getStripe(stripe_range.first_stripe);
+ const auto& name = stripe->getWriterTimezone();
+ const auto writer_timezone = name.empty() ? "GMT" : name;
+ if (!TimezoneUtils::find_cctz_time_zone(writer_timezone, _state->writer_timezone_obj)) {
+ return Status::InvalidArgument("Unknown ORC stripe writer timezone: {}",
+ writer_timezone);
+ }
+ // Matching zones preserves the SDK's UTC carrier until its negative-fraction
+ // correction is complete. cctz then applies historical offsets to that original second.
+ _state->row_reader_options.setTimezoneName(writer_timezone);
Review Comment:
[P2] Keep ORC MIN/MAX consistent with the repaired row timestamps. With a C++ ORC New York file containing an ordinary timestamp at source second `-5364644638` and fraction `123456789`, plus a hidden NANOS sibling, this matching-zone path now correctly returns `1800-01-01 00:00:00.123457`. However, [the pinned writer's statistics](https://github.com/apache/doris-thirdparty/blob/5860f94768b94406f31698ac66f4cc3d2a781324/c%2B%2B/src/ColumnWriter.cc#L1713-L1722) use ORC's ancient EST offset, and `get_aggregate_result` still interprets those bounds in UTC: pushed `MIN(plain)`/`MAX(plain)` return `1799-12-31 23:56:02.123457`. The nonzero nanos tail passes the sentinel fallback. This is a distinct metadata-aggregate path from the earlier row-carrier thread; negative timestamp predicates already remain residual. Reconcile these ordinary timestamp statistics with the selected producer/carrier semantics or fall back to row aggregation, and add a pushdown-enabled/disabled differential test.
##########
be/src/core/data_type_serde/data_type_datetimev2_serde.cpp:
##########
@@ -1049,7 +1050,11 @@ Status DataTypeDateTimeV2SerDe::read_column_from_orc(IColumn& column,
if (orc_serde_utils::orc_decode_row_count(view.rows, view.selected_rows) == 0) {
return Status::OK();
}
- return decode_timestamp_orc_values(column, view, *view.timezone);
+ const auto* timezone = view.file_type->getKind() == ::orc::TIMESTAMP &&
+ view.plain_timestamp_timezone != nullptr
+ ? view.plain_timestamp_timezone
Review Comment:
[P2] Preserve ancient timestamps produced by the Java ORC writer. A supported non-UTC Java ORC 1.9 writer in `America/New_York` stores `Timestamp.valueOf("1800-01-01 00:00:00")` as second `-5364644400`: [Java's historical timezone carrier uses raw EST](https://github.com/openjdk/jdk17u/blob/master/src/java.base/share/classes/sun/util/calendar/ZoneInfo.java#L250-L261). With a hidden NANOS-annotated sibling, the new matching-zone reader preserves that second, and this branch applies cctz's UTC-04:56:02 LMT offset, returning `1800-01-01 00:03:58`. The prior UTC-session reader returned midnight. The [Java ORC writer](https://github.com/apache/orc/blob/rel/release-1.9.0/java/core/src/java/org/apache/orc/impl/writer/TimestampTreeWriter.java#L128-L141) and public Iceberg schema converter support this mixed file. The existing C++ ancient fixture uses a different carrier, so fixing its earlier thread does not cover this regression. Preserve both producer representations and add a Java-prod
uced fixture read in a UTC session.
##########
fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergPartitionUtils.java:
##########
@@ -801,6 +828,11 @@ static boolean isValidRelatedTable(Table table) {
return false;
}
PartitionField partitionField = fields.get(0);
+ if (spec.schema().findType(partitionField.sourceId()).typeId() == Type.TypeID.TIMESTAMP_NANO) {
Review Comment:
[P2] Handle dropped partition sources before inspecting their type. A valid v1 table can remove its sole `day(ts)` partition field, leaving `void(ts)`, and then drop `ts`. [Iceberg rebuilds historical specs against the new schema](https://github.com/apache/iceberg/blob/apache-iceberg-1.11.0/core/src/main/java/org/apache/iceberg/TableMetadata.java#L756-L768), preserving the removed source ID, so `findType(sourceId)` is null in both the old calendar spec and current void spec. This new dereference runs before rejecting non-calendar transforms. `IcebergConnectorMetadata.latestSnapshotPin` calls it unconditionally during `beginQuerySnapshot`, so even a normal SELECT of a surviving column now fails with a wrapped null-pointer error; the previous code returned an unpartitioned view. Treat a missing source type as ineligible for RANGE partitioning, and cover a real v1 remove-partition/drop-source lifecycle through snapshot planning.
##########
be/src/format_v2/orc/orc_reader.cpp:
##########
@@ -1646,24 +1715,91 @@ Status OrcReader::_select_stripe_ranges_by_statistics() {
return Status::OK();
}
-void OrcReader::_apply_current_stripe_range() {
- if (!_state->stripe_pruning_applied || _state->selected_stripe_ranges.empty()) {
- return;
+Status OrcReader::_prepare_timestamp_stripe_ranges() {
+ if (!_state->match_writer_timezone) {
+ return Status::OK();
+ }
+ std::vector<uint64_t> stripes;
+ if (_state->stripe_ranges_applied) {
+ for (const auto& range : _state->selected_stripe_ranges) {
+ for (auto stripe = range.first_stripe; stripe < range.last_stripe; ++stripe) {
+ stripes.push_back(stripe);
+ }
+ }
+ } else {
+ RETURN_IF_ERROR(_collect_split_stripes(&stripes));
+ }
+ _state->selected_stripe_ranges.clear();
+ _state->stripe_ranges_applied = true;
+ _reader_statistics.read_row_groups = cast_set<int32_t>(stripes.size());
+ if (stripes.empty()) {
+ return Status::OK();
+ }
+ try {
+ uint64_t first_row = 0;
+ size_t selected = 0;
+ for (uint64_t index = 0; index <= stripes.back(); ++index) {
+ const auto stripe = _state->reader->getStripe(index);
+ const auto end_row = first_row + stripe->getNumberOfRows();
+ if (index == stripes[selected]) {
+ // A RowReader has one timezone, whereas writer timezones belong to stripes.
+ // Read footer timezone metadata only when entering the selected stripe.
+ _state->selected_stripe_ranges.push_back({
+ .first_stripe = index,
+ .last_stripe = index + 1,
Review Comment:
[P2] Reuse row readers for adjacent stripes with the same timezone. Every selected stripe becomes its own range here, including unfiltered all-GMT nanos files, and each range transition constructs a new `RowReader`. The [pinned constructor](https://github.com/apache/doris-thirdparty/blob/5860f94768b94406f31698ac66f4cc3d2a781324/c%2B%2B/src/Reader.cc#L263-L307) allocates `firstRowOfStripe` for the entire file and copies/scans every footer stripe even for a one-stripe range. For N small flushed stripes in one split this changes O(N) setup to O(N²), plus repeated schema and batch initialization; projecting only a plain timestamp beside a hidden NANOS field also triggers it. Coalesce adjacent selected stripes with identical writer zones or retain reusable SDK setup, and cover many-stripe files while preserving different-zone correctness.
##########
fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorTransaction.java:
##########
@@ -929,14 +930,19 @@ private Expression buildPartitionFilter(Map<String, String> staticPartitions, Pa
* both paths that build this predicate: DELETE/UPDATE/MERGE conflict detection and
* {@code INSERT OVERWRITE ... PARTITION(d='nan')}.
*/
- private static Expression identityPartitionPredicate(String sourceColName, Object value) {
+ private static Expression identityPartitionPredicate(Types.NestedField sourceField, Object value) {
+ String sourceColName = sourceField.name();
if (value == null) {
return Expressions.isNull(sourceColName);
}
if ((value instanceof Double || value instanceof Float)
&& Double.isNaN(((Number) value).doubleValue())) {
return Expressions.isNaN(sourceColName);
}
+ if (sourceField.type().typeId() == TypeID.TIMESTAMP_NANO) {
+ // Partition values are already nanos; a raw Long would be rebound as microseconds.
+ return Expressions.equal(sourceColName, Expressions.nanos((Long) value));
Review Comment:
[P1] Guard generated overwrite predicates against retained DATE bounds. A v3 table can retain an unpartitioned DATE file containing `2024-01-02`, promote that field to `timestamp_ns`, then add an identity partition. `INSERT OVERWRITE ... PARTITION(ts='2024-01-02')` reaches this direct nanos equality without `forTable`'s history safeguard. Iceberg 1.11 interprets the retained four-byte bounds as the unscaled long `19724`, so [its deletion evaluator](https://github.com/apache/iceberg/blob/apache-iceberg-1.11.0/core/src/main/java/org/apache/iceberg/ManifestFilterManager.java#L580-L620) excludes the old file from deletion while the replacement is committed; the overwritten rows remain. Generated identity conflict filters use the same helper. This is a separate commit-time instance from the earlier user-predicate pruning thread. Reject unsafe static overwrites or safely promote old bounds; conflict-only filters can conservatively omit the unsafe conjunct. Cover retained files from the
previous unpartitioned spec.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](datalake) Support nanosecond timestamps in Iceberg and Paimon [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #68934:
URL: https://github.com/apache/doris/pull/68934#issuecomment-6139464704
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](datalake) Support nanosecond timestamps in Iceberg and Paimon [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on code in PR #68934:
URL: https://github.com/apache/doris/pull/68934#discussion_r4255722210
##########
fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorTransaction.java:
##########
@@ -929,14 +930,19 @@ private Expression buildPartitionFilter(Map<String, String> staticPartitions, Pa
* both paths that build this predicate: DELETE/UPDATE/MERGE conflict detection and
* {@code INSERT OVERWRITE ... PARTITION(d='nan')}.
*/
- private static Expression identityPartitionPredicate(String sourceColName, Object value) {
+ private static Expression identityPartitionPredicate(Types.NestedField sourceField, Object value) {
+ String sourceColName = sourceField.name();
if (value == null) {
return Expressions.isNull(sourceColName);
}
if ((value instanceof Double || value instanceof Float)
&& Double.isNaN(((Number) value).doubleValue())) {
return Expressions.isNaN(sourceColName);
}
+ if (sourceField.type().typeId() == TypeID.TIMESTAMP_NANO) {
+ // Partition values are already nanos; a raw Long would be rebound as microseconds.
+ return Expressions.equal(sourceColName, Expressions.nanos((Long) value));
Review Comment:
I will share the table-history safety check with generated partition predicates. Static partition overwrite must fail before committing when retained DATE bounds make the nanos predicate unsafe; conflict-only filters will conservatively omit unsafe predicates. I will cover retained unpartitioned DATE files followed by timestamp_ns and partition evolution.
##########
fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergPredicateConverter.java:
##########
@@ -475,7 +507,10 @@ private Types.NestedField getPushdownField(String colName) {
|| ICEBERG_LAST_UPDATED_SEQUENCE_NUMBER_COL.equalsIgnoreCase(colName)) {
return null;
}
- return schema.caseInsensitiveFindField(colName);
+ Types.NestedField field = schema.caseInsensitiveFindField(colName);
+ // Iceberg 1.11 widens retained four-byte DATE bounds to unscaled longs. Pruning those
+ // epoch-day bounds with nanosecond predicates can discard matching files in every planner.
+ return field != null && unsafeNanosFields.contains(field.fieldId()) ? null : field;
Review Comment:
I will make NOT conversion require an exact conversion of its entire operand. Dropping an unsafe nanos conjunct is valid for a positive scan filter but becomes incorrect after negation. I will add coverage for NOT over a partially convertible AND and verify that matching files are retained.
##########
fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergPartitionUtils.java:
##########
@@ -801,6 +828,11 @@ static boolean isValidRelatedTable(Table table) {
return false;
}
PartitionField partitionField = fields.get(0);
+ if (spec.schema().findType(partitionField.sourceId()).typeId() == Type.TypeID.TIMESTAMP_NANO) {
Review Comment:
I will treat a missing historical partition source type as an unsupported partition projection, rather than dereferencing it. This is an Iceberg metadata issue (including v1 tables), not the excluded Doris format-v1 reader path. I will cover removing a partition field and then dropping its source column.
##########
be/src/core/data_type_serde/data_type_datetimev2_serde.cpp:
##########
@@ -1049,7 +1050,11 @@ Status DataTypeDateTimeV2SerDe::read_column_from_orc(IColumn& column,
if (orc_serde_utils::orc_decode_row_count(view.rows, view.selected_rows) == 0) {
return Status::OK();
}
- return decode_timestamp_orc_values(column, view, *view.timezone);
+ const auto* timezone = view.file_type->getKind() == ::orc::TIMESTAMP &&
+ view.plain_timestamp_timezone != nullptr
+ ? view.plain_timestamp_timezone
Review Comment:
I will investigate and fix the historical plain-TIMESTAMP interpretation on the Doris side using producer-aware ORC timestamp semantics, without modifying the SDK. Validation will distinguish Java raw-offset behavior from the C++ historical writer-timezone behavior.
##########
be/src/format_v2/orc/orc_reader.cpp:
##########
@@ -1646,24 +1715,91 @@ Status OrcReader::_select_stripe_ranges_by_statistics() {
return Status::OK();
}
-void OrcReader::_apply_current_stripe_range() {
- if (!_state->stripe_pruning_applied || _state->selected_stripe_ranges.empty()) {
- return;
+Status OrcReader::_prepare_timestamp_stripe_ranges() {
+ if (!_state->match_writer_timezone) {
+ return Status::OK();
+ }
+ std::vector<uint64_t> stripes;
+ if (_state->stripe_ranges_applied) {
+ for (const auto& range : _state->selected_stripe_ranges) {
+ for (auto stripe = range.first_stripe; stripe < range.last_stripe; ++stripe) {
+ stripes.push_back(stripe);
+ }
+ }
+ } else {
+ RETURN_IF_ERROR(_collect_split_stripes(&stripes));
+ }
+ _state->selected_stripe_ranges.clear();
+ _state->stripe_ranges_applied = true;
+ _reader_statistics.read_row_groups = cast_set<int32_t>(stripes.size());
+ if (stripes.empty()) {
+ return Status::OK();
+ }
+ try {
+ uint64_t first_row = 0;
+ size_t selected = 0;
+ for (uint64_t index = 0; index <= stripes.back(); ++index) {
+ const auto stripe = _state->reader->getStripe(index);
+ const auto end_row = first_row + stripe->getNumberOfRows();
+ if (index == stripes[selected]) {
+ // A RowReader has one timezone, whereas writer timezones belong to stripes.
+ // Read footer timezone metadata only when entering the selected stripe.
+ _state->selected_stripe_ranges.push_back({
+ .first_stripe = index,
+ .last_stripe = index + 1,
+ .offset = stripe->getOffset(),
+ .length = stripe->getLength(),
+ .first_row = first_row,
+ .end_row = end_row,
+ });
+ ++selected;
+ }
+ first_row = end_row;
+ }
+ } catch (const std::exception& e) {
+ return Status::InternalError("Failed to prepare ORC timestamp stripe ranges: {}", e.what());
+ }
+ return Status::OK();
+}
+
+Status OrcReader::_apply_current_stripe_range() {
+ if (!_state->stripe_ranges_applied || _state->selected_stripe_ranges.empty()) {
+ return Status::OK();
}
DORIS_CHECK(_state->current_stripe_range < _state->selected_stripe_ranges.size());
const auto& stripe_range = _state->selected_stripe_ranges[_state->current_stripe_range];
_state->row_reader_options.range(stripe_range.offset, stripe_range.length);
+ if (_state->match_writer_timezone) {
+ try {
+ const auto stripe = _state->reader->getStripe(stripe_range.first_stripe);
+ const auto& name = stripe->getWriterTimezone();
+ const auto writer_timezone = name.empty() ? "GMT" : name;
+ if (!TimezoneUtils::find_cctz_time_zone(writer_timezone, _state->writer_timezone_obj)) {
+ return Status::InvalidArgument("Unknown ORC stripe writer timezone: {}",
+ writer_timezone);
+ }
+ // Matching zones preserves the SDK's UTC carrier until its negative-fraction
+ // correction is complete. cctz then applies historical offsets to that original second.
+ _state->row_reader_options.setTimezoneName(writer_timezone);
Review Comment:
I will make metadata MIN/MAX fall back to row decoding where the ORC timestamp statistics cannot preserve the same historical civil-time semantics as the reader. I will add aggregate-versus-row coverage; no SDK change is planned.
##########
be/src/format_v2/orc/orc_reader.cpp:
##########
@@ -1646,24 +1715,91 @@ Status OrcReader::_select_stripe_ranges_by_statistics() {
return Status::OK();
}
-void OrcReader::_apply_current_stripe_range() {
- if (!_state->stripe_pruning_applied || _state->selected_stripe_ranges.empty()) {
- return;
+Status OrcReader::_prepare_timestamp_stripe_ranges() {
+ if (!_state->match_writer_timezone) {
+ return Status::OK();
+ }
+ std::vector<uint64_t> stripes;
+ if (_state->stripe_ranges_applied) {
+ for (const auto& range : _state->selected_stripe_ranges) {
+ for (auto stripe = range.first_stripe; stripe < range.last_stripe; ++stripe) {
+ stripes.push_back(stripe);
+ }
+ }
+ } else {
+ RETURN_IF_ERROR(_collect_split_stripes(&stripes));
+ }
+ _state->selected_stripe_ranges.clear();
+ _state->stripe_ranges_applied = true;
+ _reader_statistics.read_row_groups = cast_set<int32_t>(stripes.size());
+ if (stripes.empty()) {
+ return Status::OK();
+ }
+ try {
+ uint64_t first_row = 0;
+ size_t selected = 0;
+ for (uint64_t index = 0; index <= stripes.back(); ++index) {
+ const auto stripe = _state->reader->getStripe(index);
+ const auto end_row = first_row + stripe->getNumberOfRows();
+ if (index == stripes[selected]) {
+ // A RowReader has one timezone, whereas writer timezones belong to stripes.
+ // Read footer timezone metadata only when entering the selected stripe.
+ _state->selected_stripe_ranges.push_back({
+ .first_stripe = index,
+ .last_stripe = index + 1,
Review Comment:
I will coalesce adjacent selected stripes that use the same effective timestamp reader timezone, while preserving split boundaries, pruning gaps, row IDs, and timezone changes. This avoids recreating an SDK RowReader for every stripe without modifying the SDK.
##########
be/src/format_v2/parquet/native_schema_desc.cpp:
##########
@@ -882,6 +882,11 @@ std::pair<DataTypePtr, bool> NativeFieldDescriptor::convert_to_doris_type(
// sub-second values are silently truncated by the target SerDe.
ans.first = DataTypeFactory::instance().create_data_type(TYPE_TIMEV2, nullable, 0, scale);
} else if (logicalType.__isset.TIMESTAMP) {
+ if (logicalType.TIMESTAMP.unit.__isset.NANOS && !logicalType.TIMESTAMP.isAdjustedToUTC) {
+ // Materialize nanoseconds before any table cast; DATETIMEV2 would discard the remainder.
+ ans.first = DataTypeFactory::instance().create_data_type(TYPE_TIMESTAMP_NS, nullable);
Review Comment:
I will add ColumnTimeStampNs to the native nullable-column dispatch so filtered nullable nanos pages use the fused decoding path. Tests will check both decoded values and the fallback counter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](datalake) Support nanosecond timestamps in Iceberg and Paimon [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68934:
URL: https://github.com/apache/doris/pull/68934#issuecomment-6125217832
<details>
<summary>ClickBench: <b>Total hot run time: 24.84 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 643eb975c593ce69a4c3592b6a5769e053cb6b6f, data reload: false
query1 0.00 0.00 0.01
query2 0.10 0.05 0.05
query3 0.27 0.12 0.12
query4 1.61 0.14 0.13
query5 0.28 0.25 0.22
query6 1.17 0.70 0.68
query7 0.04 0.01 0.01
query8 0.05 0.03 0.03
query9 0.43 0.36 0.36
query10 0.56 0.61 0.58
query11 0.21 0.15 0.15
query12 0.18 0.15 0.15
query13 0.49 0.50 0.50
query14 1.00 0.96 0.97
query15 0.63 0.62 0.60
query16 0.35 0.35 0.36
query17 1.15 1.16 1.13
query18 0.23 0.22 0.21
query19 2.07 1.98 1.99
query20 0.02 0.01 0.02
query21 15.42 0.28 0.16
query22 4.42 0.06 0.05
query23 16.26 0.33 0.12
query24 3.16 0.55 0.76
query25 0.14 0.05 0.06
query26 1.77 0.24 0.16
query27 0.03 0.04 0.05
query28 2.87 0.62 0.42
query29 12.51 4.18 3.32
query30 0.29 0.15 0.14
query31 2.76 0.58 0.35
query32 3.23 0.63 0.52
query33 3.29 3.30 3.23
query34 15.61 4.00 3.39
query35 3.34 3.37 3.35
query36 0.56 0.44 0.42
query37 0.10 0.07 0.06
query38 0.05 0.04 0.04
query39 0.04 0.03 0.03
query40 0.17 0.15 0.15
query41 0.09 0.03 0.02
query42 0.03 0.03 0.02
query43 0.04 0.04 0.03
Total cold run time: 97.02 s
Total hot run time: 24.84 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](datalake) Support nanosecond timestamps in Iceberg and Paimon [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #68934:
URL: https://github.com/apache/doris/pull/68934#issuecomment-6136204449
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](datalake) Support nanosecond timestamps in Iceberg and Paimon [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #68934:
URL: https://github.com/apache/doris/pull/68934#discussion_r4246410050
##########
fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergSchemaBuilder.java:
##########
@@ -360,10 +362,16 @@ public static Literal<?> parseDefaultLiteral(String value, Type type) {
} catch (NumberFormatException e) {
throw new IllegalArgumentException("Invalid Int string: " + value, e);
}
+ case TIMESTAMP_NANO:
+ // ADD COLUMN carries the validated SQL datetime, not an epoch integer.
+ if (!value.matches("[+-]?[0-9]+")) {
Review Comment:
[P2] Parse compact SQL defaults as civil timestamps. `ALTER TABLE t ADD COLUMN ts TIMESTAMP_NS DEFAULT '20240102'` passes FE validation as 2024-01-02 midnight, and the original text reaches this method unchanged. This numeric branch instead commits `Expressions.nanos(20240102L)`, making both initial and write defaults `1970-01-01 00:00:00.020240102`. Normalize the validated temporal literal or parse its SQL formats here, and cover compact date/datetime defaults. This is a different outcome from the earlier canonical-text rejection: the new discriminator succeeds with the wrong persisted value.
##########
be/src/core/data_type_serde/orc_serde_utils.cpp:
##########
@@ -23,6 +23,32 @@
namespace doris::orc_serde_utils {
+bool has_nanosecond_timestamp(const ::orc::Type& type) {
+ if (type.getKind() == ::orc::TIMESTAMP && type.hasAttributeKey("iceberg.timestamp-unit") &&
+ type.getAttributeValue("iceberg.timestamp-unit") == "NANOS") {
+ return true;
+ }
+ for (uint64_t child = 0; child < type.getSubtypeCount(); ++child) {
+ if (has_nanosecond_timestamp(*type.getSubtype(child))) {
+ return true;
+ }
+ }
+ return false;
+}
+
+int64_t corrected_orc_timestamp_seconds(const OrcDecodedColumnView& view, int64_t seconds,
+ int64_t nanoseconds) {
+ if (view.writer_epoch_offset == 0 || nanoseconds <= 999999 ||
Review Comment:
[P2] Preserve sparse follower timestamps in the GMT carrier. The [pinned ORC selected decoder](https://github.com/apache/doris-thirdparty/blob/5860f94768b94406f31698ac66f4cc3d2a781324/c%2B%2B/src/ColumnReader.cc#L405-L431) tests `nanoBuffer[i]` instead of `nanoBuffer[idx]`. For a New York writer with plain timestamp seconds `[1,1]`, fractions `[500000123,1]`, an unprojected NANOS sibling and an integer predicate selecting row 1, GMT shifts the carrier negative. The SDK tests row 0's packed fraction and wrongly subtracts a second; this guard preserves it. The plain result becomes `1969-12-31 19:00:00.000000` instead of `19:00:01.000000`. Dense decoding and the prior matching-zone reader are correct. Fix or bypass that selected-index path and test lazy/dense equivalence; this differs from the existing dense epoch-zero issue.
##########
fe/be-java-extensions/paimon-scanner/src/main/java/org/apache/doris/paimon/PaimonArrowBatchAdapter.java:
##########
@@ -204,6 +204,10 @@ private static FieldType timestampField(
TimeUnit unit = precision > 3
? TimeUnit.MICROSECOND
: precision > 0 ? TimeUnit.MILLISECOND : TimeUnit.SECOND;
+ if (timeZone == null && precision > 6) {
+ // The transport must retain the sub-microsecond digits of TIMESTAMP_NS inputs.
+ unit = TimeUnit.NANOSECOND;
Review Comment:
[P2] Normalize nanoseconds to the Paimon target precision. Existing Paimon `TIMESTAMP(7/8)` fields now map to `TIMESTAMP_NS`, so inserting `'2024-01-02 03:04:05.123456789'` retains all nine digits in the sink. This branch emits nanoseconds, `timestampAdapter.getTimestamp` ignores target precision, and `PaimonWriteSchema.tableRow` copies the value unchanged. Paimon's serializer/INT96 writer therefore persists `.123456789` instead of `.123456700`/`.123456780`, also changing timestamp-key equality across writers. Apply target precision normalization, including nested fields, and test writes to existing precision-7/8 schemas. Paimon's [timestamp cast](https://github.com/apache/paimon/blob/release-1.4.2/paimon-common/src/main/java/org/apache/paimon/casting/TimestampToTimestampCastRule.java#L72-L75) supplies the expected narrowing semantics.
##########
be/src/core/data_type_serde/orc_serde_utils.cpp:
##########
@@ -23,6 +23,32 @@
namespace doris::orc_serde_utils {
+bool has_nanosecond_timestamp(const ::orc::Type& type) {
+ if (type.getKind() == ::orc::TIMESTAMP && type.hasAttributeKey("iceberg.timestamp-unit") &&
+ type.getAttributeValue("iceberg.timestamp-unit") == "NANOS") {
+ return true;
+ }
+ for (uint64_t child = 0; child < type.getSubtypeCount(); ++child) {
+ if (has_nanosecond_timestamp(*type.getSubtype(child))) {
+ return true;
+ }
+ }
+ return false;
+}
+
+int64_t corrected_orc_timestamp_seconds(const OrcDecodedColumnView& view, int64_t seconds,
+ int64_t nanoseconds) {
+ if (view.writer_epoch_offset == 0 || nanoseconds <= 999999 ||
+ view.file_type->getKind() != ::orc::TIMESTAMP) {
+ return seconds;
+ }
+ // ORC 1.9 applies the negative-fraction correction after changing timezones. Undo it
+ // in the GMT carrier and apply it at the writer's epoch boundary instead, preserving
+ // fractions when the timezone shift crosses zero. TIMESTAMP_INSTANT is always UTC.
+ const int64_t encoded_seconds = seconds < 0 ? seconds + 1 : seconds;
Review Comment:
[P2] Restore the writer second before applying timezone rules. For an external New York stripe, `1969-10-26 01:59:59.500000000` has writer second `-5767201`. ORC increments it for fractional encoding; the new GMT reader then selects UTC-05 at `-5767200`, after the fallback, rather than UTC-04. This helper cannot recover the lost offset and returns `00:59:59.500000000`; the prior matching-zone reader returned `01:59:59.500000000`. An unprojected nanosecond sibling also triggers this. Correct timezone conversion in the original writer-second domain and cover a historical transition. Unlike the existing constant-offset epoch-zero finding, this loses a timezone rule. See the [pinned conversion order](https://github.com/apache/doris-thirdparty/blob/5860f94768b94406f31698ac66f4cc3d2a781324/c%2B%2B/src/ColumnReader.cc#L370-L386).
##########
be/src/core/data_type_serde/data_type_timestamp_ns_serde.cpp:
##########
@@ -345,6 +398,112 @@ Status DataTypeTimeStampNsSerDe::deserialize_column_from_jsonb_vector(
source_null_map);
}
+Status DataTypeTimeStampNsSerDe::read_parquet_dictionary(
+ IColumn& column, ParquetDecodeSource& source, const ParquetDecodeContext& context) const {
+ TimestampNsParquetConsumer consumer(column, context);
+ RejectTimestampNsBinaryConsumer binary;
+ return source.decode_dictionary(consumer, binary);
+}
+
+Status DataTypeTimeStampNsSerDe::read_column_from_parquet(
+ IColumn& column, ParquetDecodeSource& source, const ParquetDecodeContext& context,
+ size_t num_values, ParquetMaterializationState& state) const {
+ TimestampNsParquetConsumer consumer(column, context, &state);
+ if (context.encoding != ParquetValueEncoding::DICTIONARY) {
+ return source.decode_fixed_values(num_values, consumer);
+ }
+ if (state.dictionary_generation != source.dictionary_generation()) {
+ state.typed_dictionary = column.clone_empty();
+ auto* null_map = state.begin_dictionary_conversion(source.dictionary_size());
+ TimestampNsParquetConsumer dictionary(*state.typed_dictionary, context, &state);
+ RejectTimestampNsBinaryConsumer binary;
+ auto status = source.decode_dictionary(dictionary, binary);
+ state.end_dictionary_conversion(null_map);
+ RETURN_IF_ERROR(status);
+ DORIS_CHECK_EQ(state.typed_dictionary->size(), source.dictionary_size());
+ state.dictionary_generation = source.dictionary_generation();
+ }
+ return state.materialize_dictionary(column, source, num_values);
+}
+
+Status DataTypeTimeStampNsSerDe::read_column_from_decoded_values(
+ IColumn& column, const DecodedColumnView& view) const {
+ if (view.value_kind != DecodedValueKind::INT64 && view.value_kind != DecodedValueKind::INT96) {
+ return Status::NotSupported("TIMESTAMP_NS expects decoded INT64 or INT96 values");
+ }
+ if (view.values == nullptr && decoded_column_view_has_non_null_value(view)) {
+ return Status::Corruption("Decoded TIMESTAMP_NS value buffer is null");
+ }
+ auto unit = ParquetTimeUnit::MICROS;
+ if (view.time_unit == DecodedTimeUnit::MILLIS) {
+ unit = ParquetTimeUnit::MILLIS;
+ } else if (view.time_unit == DecodedTimeUnit::NANOS) {
+ unit = ParquetTimeUnit::NANOS;
+ }
+ auto& data = assert_cast<ColumnTimeStampNs&>(column).get_data();
+ const auto old_size = data.size();
+ for (int64_t row = 0; row < view.row_count; ++row) {
+ if (decoded_column_view_row_is_null(view, row)) {
+ data.emplace_back();
+ continue;
+ }
+ int64_t nanos;
+ auto status =
+ view.value_kind == DecodedValueKind::INT96
+ ? parquet_int96_timestamp_nanos(
+ unaligned_load<ParquetInt96Timestamp>(
+ view.values + row * sizeof(ParquetInt96Timestamp)),
+ &nanos)
+ : parquet_timestamp_nanos(
+ unit,
+ unaligned_load<int64_t>(view.values + row * sizeof(int64_t)),
+ &nanos);
+ if (!status.ok()) {
+ if (decoded_column_view_can_null_on_conversion_failure(view)) {
+ decoded_column_view_insert_null_on_conversion_failure(column, view, row);
+ continue;
+ }
+ data.resize(old_size);
+ return status;
+ }
+ data.emplace_back(nanos);
+ }
+ return Status::OK();
+}
+
+Status DataTypeTimeStampNsSerDe::read_column_from_orc(IColumn& column,
+ const OrcDecodedColumnView& view) const {
+ const auto& batch = assert_cast<const orc::TimestampVectorBatch&>(*view.batch);
+ auto& data = assert_cast<ColumnTimeStampNs&>(column).get_data();
+ const auto old_size = data.size();
+ const auto rows = orc_serde_utils::orc_decode_row_count(view.rows, view.selected_rows);
+ const auto timezone = view.timezone == nullptr ? cctz::utc_time_zone() : *view.timezone;
+ for (size_t row = 0; row < rows; ++row) {
+ const auto source = orc_serde_utils::orc_source_row_at(row, view.selected_rows);
+ if (orc_serde_utils::orc_row_is_null(batch, source)) {
+ data.emplace_back();
+ continue;
+ }
+ const auto nanos = batch.nanoseconds[source];
+ if (nanos < 0 || nanos >= TimeStampNsValue::NANOS_PER_SECOND) {
+ data.resize(old_size);
+ return Status::DataQualityError("Invalid ORC timestamp nanoseconds: {}", nanos);
+ }
+ DateV2Value<DateTimeV2ValueType> civil;
+ const int64_t seconds =
+ orc_serde_utils::corrected_orc_timestamp_seconds(view, batch.data[source], nanos);
+ auto status = orc_serde_utils::orc_timestamp_to_datetime(seconds, nanos / 1000, timezone,
Review Comment:
[P2] Preserve ancient offsets from the C++ ORC UTC carrier. The [pinned C++ API](https://github.com/apache/doris-thirdparty/blob/5860f94768b94406f31698ac66f4cc3d2a781324/c%2B%2B/include/orc/Vector.hh#L403-L406) requires caller-converted UTC seconds. A New York writer can therefore store `1800-01-01 00:00:00` as `-5364644638`, with zero nanos and the historical UTC-04:56:02 offset. File-wide GMT reading uses ORC's ancient EST offset instead, so this conversion returns `1799-12-31 23:56:02`. The prior matching-zone reader preserved the correct ordinary timestamp in the same mixed file. An unprojected NANOS sibling also activates the regression. Preserve that supported writer carrier and add an external C++ historical fixture; fractional-second and sparse-index fixes cannot repair this whole-second case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](datalake) Support nanosecond timestamps in Iceberg and Paimon [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on code in PR #68934:
URL: https://github.com/apache/doris/pull/68934#discussion_r4254161930
##########
fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergPredicateConverter.java:
##########
@@ -671,6 +679,19 @@ private long toMicros(LocalDateTime dateTime, Type icebergType, boolean utcCarri
return instant.getEpochSecond() * 1_000_000L + instant.getNano() / 1000L;
}
+ private static Literal<Long> toNanos(LocalDateTime dateTime, Type icebergType) {
+ if (((Types.TimestampNanoType) icebergType).shouldAdjustToUTC()) {
+ return null;
+ }
+ try {
+ // A raw Long is a microsecond literal in Iceberg; preserve the nanosecond unit explicitly.
+ // Unzoned values use a UTC wall-clock carrier, never the session zone.
+ return Expressions.nanos(IcebergPartitionUtils.toEpochNanos(dateTime));
Review Comment:
Fixed the metadata-cleanup lifecycle gap in fd85964c3b4. The converter now treats gaps in Iceberg schema IDs as incomplete history and conservatively suppresses nanos predicates; retained history is captured once for the check. Added actual snapshot expiration with cleanExpiredMetadata(true), verifying all converter modes and SDK/manifest-cache planning preserve the retained DATE file. Complete history still permits fields without DATE promotion.
##########
fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergTypeMapping.java:
##########
@@ -150,12 +150,16 @@ private static ConnectorType fromPrimitive(Type.PrimitiveType primitive,
return ConnectorType.of("TIMESTAMPTZ", ICEBERG_DATETIME_SCALE_MS, 0);
}
return ConnectorType.of("DATETIMEV2", ICEBERG_DATETIME_SCALE_MS, 0);
+ case TIMESTAMP_NANO:
+ // TIMESTAMP_NS is a wall-clock type; zoned nanoseconds need a distinct carrier.
+ return ConnectorType.of(((Types.TimestampNanoType) primitive).shouldAdjustToUTC()
+ ? "UNSUPPORTED" : "TIMESTAMP_NS");
Review Comment:
Fixed the remaining file-filter path in fd85964c3b4. Localized file-to-table casts now carry the schema-evolution flag, so non-strict SQL settings cannot turn DATE overflow into NULL before IS NOT NULL filtering. The real Parquet reader test failed before the fix and now passes with filtered/unfiltered scans and strict/non-strict settings. The related 371 BE tests pass.
##########
fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergPartitionUtils.java:
##########
@@ -479,6 +491,13 @@ static Object parsePartitionValueFromString(String valueStr, Type icebergType, Z
case DATE:
// Iceberg date is days since epoch (1970-01-01).
return (int) LocalDate.parse(valueStr, DateTimeFormatter.ISO_LOCAL_DATE).toEpochDay();
+ case TIMESTAMP_NANO:
+ LocalDateTime timestampNs = LocalDateTime.parse(valueStr.replace('T', ' '),
Review Comment:
Fixed the explicit-offset static-partition case in fd85964c3b4. Nanosecond partition parsing now converts explicit offsets to session-local civil time, matching the sink cast. Tests cover UTC, Asia/Shanghai and America/New_York with nine-digit fractions, plus a real Iceberg overwrite retaining the adjacent partition. The Parquet and ORC regression suite passed locally with the final FE/BE builds; expected output was generated by actual execution. All 325 related FE tests pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](datalake) Support nanosecond timestamps in Iceberg and Paimon [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on code in PR #68934:
URL: https://github.com/apache/doris/pull/68934#discussion_r4256383241
##########
fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorTransaction.java:
##########
@@ -929,14 +930,19 @@ private Expression buildPartitionFilter(Map<String, String> staticPartitions, Pa
* both paths that build this predicate: DELETE/UPDATE/MERGE conflict detection and
* {@code INSERT OVERWRITE ... PARTITION(d='nan')}.
*/
- private static Expression identityPartitionPredicate(String sourceColName, Object value) {
+ private static Expression identityPartitionPredicate(Types.NestedField sourceField, Object value) {
+ String sourceColName = sourceField.name();
if (value == null) {
return Expressions.isNull(sourceColName);
}
if ((value instanceof Double || value instanceof Float)
&& Double.isNaN(((Number) value).doubleValue())) {
return Expressions.isNaN(sourceColName);
}
+ if (sourceField.type().typeId() == TypeID.TIMESTAMP_NANO) {
+ // Partition values are already nanos; a raw Long would be rebound as microseconds.
+ return Expressions.equal(sourceColName, Expressions.nanos((Long) value));
Review Comment:
Rechecked at `19ef150201f4c4928c97b1efbe6003db6b83caf3`: this FE issue is addressed. Scan, static overwrite and generated conflict predicates now share `unsafeNanosFields`. Static overwrite rejects an unsafe non-null partition predicate without widening deletion; conflict-only filtering conservatively omits unsafe conjuncts. The added retained-DATE-file transaction test checks that rejection leaves the snapshot unchanged. This is safe rejection of the unsupported overwrite, rather than support for completing it. Source and test assertions inspected; I did not independently run the tests.
##########
fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergPartitionUtils.java:
##########
@@ -801,6 +828,11 @@ static boolean isValidRelatedTable(Table table) {
return false;
}
PartitionField partitionField = fields.get(0);
+ if (spec.schema().findType(partitionField.sourceId()).typeId() == Type.TypeID.TIMESTAMP_NANO) {
Review Comment:
Rechecked at `19ef150201f4c4928c97b1efbe6003db6b83caf3`: this FE issue is addressed. A missing historical source type now makes `isValidRelatedTable` return false, preserving the UNPARTITIONED view used by query snapshot planning. The new test exercises an actual v1 remove-partition/drop-source lifecycle. The null check handles the concrete reachable metadata state identified here. Source and test assertions inspected; I did not independently run the tests.
##########
fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonTypeMapping.java:
##########
@@ -156,6 +156,10 @@ private static ConnectorType toDecimalType(DecimalType type) {
private static ConnectorType toTimestampType(DataType dataType) {
int scale = DEFAULT_TIMESTAMP_SCALE;
if (dataType instanceof TimestampType) {
+ // Keep all source fractional digits instead of truncating high-precision wall-clock values.
+ if (((TimestampType) dataType).getPrecision() > MAX_TIMESTAMP_SCALE) {
+ return ConnectorType.of("TIMESTAMP_NS");
Review Comment:
Rechecked at `19ef150201f4c4928c97b1efbe6003db6b83caf3`: the mapping and out-of-range failure remain unchanged. I understand the stated decision to leave range compatibility outside this PR; this thread should therefore remain a known limitation rather than be marked fixed. Please explicitly disclose in the PR description/release note and type-mapping documentation that existing Paimon TIMESTAMP(7/8/9) columns now use the signed Int64 nanosecond range (approximately 1677–2262), and previously readable values outside that range fail queries. I found no new distinct FE issue in the latest delta. Static review only; I did not independently run tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](datalake) Support nanosecond timestamps in Iceberg and Paimon [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68934:
URL: https://github.com/apache/doris/pull/68934#issuecomment-6121326985
<details>
<summary>TPC-DS: <b>Total hot run time: 151992 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5024d4e894591b18ef797542542aa1b040d73696, data reload: false
query5 4310 591 461 461
query6 418 202 198 198
query7 4795 437 221 221
query8 316 180 162 162
query9 8747 3948 3957 3948
query10 501 310 257 257
query11 5910 3426 3210 3210
query12 140 90 84 84
query13 1244 426 323 323
query14 6529 4781 4492 4492
query14_1 4221 4228 4234 4228
query15 227 207 188 188
query16 955 438 408 408
query17 860 681 551 551
query18 2416 426 301 301
query19 202 158 135 135
query20 82 80 85 80
query21 212 132 110 110
query22 12994 13095 13008 13008
query23 13109 12456 12055 12055
query23_1 12338 12305 12277 12277
query24 7092 823 476 476
query24_1 483 474 469 469
query25 534 424 367 367
query26 1306 256 144 144
query27 2746 444 278 278
query28 4572 1957 1940 1940
query29 1570 610 452 452
query30 296 222 195 195
query31 896 773 647 647
query32 147 95 94 94
query33 516 295 252 252
query34 874 848 500 500
query35 746 768 681 681
query36 808 812 736 736
query37 130 103 91 91
query38 1803 1746 1678 1678
query39 707 724 692 692
query39_1 692 688 669 669
query40 217 117 101 101
query41 73 70 69 69
query42 86 86 84 84
query43 365 380 328 328
query44 1313 715 709 709
query45 181 180 170 170
query46 832 975 566 566
query47 3026 3029 2853 2853
query48 307 309 252 252
query49 563 379 307 307
query50 676 283 207 207
query51 10388 10064 10082 10064
query52 74 82 66 66
query53 185 205 148 148
query54 247 202 185 185
query55 70 70 64 64
query56 244 204 200 200
query57 1627 1598 1572 1572
query58 283 262 258 258
query59 2347 2379 2104 2104
query60 272 241 204 204
query61 147 148 145 145
query62 405 339 290 290
query63 200 158 165 158
query64 2672 945 759 759
query65 3458 3356 3376 3356
query66 1765 411 298 298
query67 20261 20133 20130 20130
query68 2928 995 617 617
query69 400 287 242 242
query70 902 838 843 838
query71 287 223 214 214
query72 2659 2513 2242 2242
query73 517 515 289 289
query74 4585 4491 4310 4310
query75 2308 2289 1968 1968
query76 2300 1015 588 588
query77 362 387 302 302
query78 9032 9055 8584 8584
query79 979 891 515 515
query80 512 424 353 353
query81 521 316 281 281
query82 581 133 109 109
query83 296 203 177 177
query84 305 122 105 105
query85 794 403 353 353
query86 309 239 230 230
query87 2018 1948 1827 1827
query88 3592 2699 2685 2685
query89 351 312 262 262
query90 1913 178 177 177
query91 157 143 120 120
query92 95 89 90 89
query93 992 989 566 566
query94 438 299 244 244
query95 591 341 387 341
query96 670 523 238 238
query97 2443 2410 2317 2317
query98 154 155 141 141
query99 781 760 646 646
Total cold run time: 233249 ms
Total hot run time: 151992 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](datalake) Support nanosecond timestamps in Iceberg and Paimon [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #68934:
URL: https://github.com/apache/doris/pull/68934#issuecomment-6138191111
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](datalake) Support nanosecond timestamps in Iceberg and Paimon [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on code in PR #68934:
URL: https://github.com/apache/doris/pull/68934#discussion_r4255472271
##########
fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorTransaction.java:
##########
@@ -929,14 +930,19 @@ private Expression buildPartitionFilter(Map<String, String> staticPartitions, Pa
* both paths that build this predicate: DELETE/UPDATE/MERGE conflict detection and
* {@code INSERT OVERWRITE ... PARTITION(d='nan')}.
*/
- private static Expression identityPartitionPredicate(String sourceColName, Object value) {
+ private static Expression identityPartitionPredicate(Types.NestedField sourceField, Object value) {
+ String sourceColName = sourceField.name();
if (value == null) {
return Expressions.isNull(sourceColName);
}
if ((value instanceof Double || value instanceof Float)
&& Double.isNaN(((Number) value).doubleValue())) {
return Expressions.isNaN(sourceColName);
}
+ if (sourceField.type().typeId() == TypeID.TIMESTAMP_NANO) {
+ // Partition values are already nanos; a raw Long would be rebound as microseconds.
+ return Expressions.equal(sourceColName, Expressions.nanos((Long) value));
Review Comment:
Confirmed in the FE review at `3f7020a33f405beaf37e5adbf07ccbb55a06774a`. `commitStaticPartitionOverwrite` builds this predicate directly, while only user-supplied constraints pass through `IcebergPredicateConverter.forTable`. The generated partition conflict filter also calls this helper. Please share the table-history safety check across these entry points: reject an unsafe static overwrite or interpret the retained bounds correctly; dropping the overwrite predicate would incorrectly broaden deletion. The regression case should retain an unpartitioned DATE file across promotion and identity-partition addition, then verify that static overwrite removes the intended old rows. Static source/dependency inspection only; I did not run tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](datalake) Support nanosecond timestamps in Iceberg and Paimon [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #68934:
URL: https://github.com/apache/doris/pull/68934#discussion_r4255653607
##########
fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergPredicateConverter.java:
##########
@@ -475,7 +507,10 @@ private Types.NestedField getPushdownField(String colName) {
|| ICEBERG_LAST_UPDATED_SEQUENCE_NUMBER_COL.equalsIgnoreCase(colName)) {
return null;
}
- return schema.caseInsensitiveFindField(colName);
+ Types.NestedField field = schema.caseInsensitiveFindField(colName);
+ // Iceberg 1.11 widens retained four-byte DATE bounds to unscaled longs. Pruning those
+ // epoch-day bounds with nanosecond predicates can discard matching files in every planner.
+ return field != null && unsafeNanosFields.contains(field.fieldId()) ? null : field;
Review Comment:
[P2] Keep unsafe nanos suppression conservative beneath NOT
With `disable_nereids_expression_rules='SIMPLIFY_NOT_EXPR'`, a supported query retains this plan:
```text
Filter[NOT(ts='2024-01-02' AND id>0)]
IcebergScan(ts promoted from DATE, id)
```
For a v3 unpartitioned table retaining a DATE file with `ts='2024-01-03'` and `id=1`, this row should survive. The new unsafe-field suppression drops `ts` in `getPushdownField`; SCAN `buildAnd` keeps `id>0`, then `buildNot` produces `NOT(id>0)`. Iceberg rewrites that to `id<=0`, and its [metrics evaluator](https://github.com/apache/iceberg/blob/apache-iceberg-1.11.0/api/src/main/java/org/apache/iceberg/expressions/InclusiveMetricsEvaluator.java#L229-L249) rejects the file with lower bound `id=1`. The residual cannot recover the missing row. Drop the whole negated subtree when any child conversion is inexact, or track conversion exactness through NOT, and cover SDK/cached planning with this rule disabled. This differs from the existing DATE-bound thread: it prunes on `id` after suppressing the unsafe timestamp comparison.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](datalake) Support nanosecond timestamps in Iceberg and Paimon [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on code in PR #68934:
URL: https://github.com/apache/doris/pull/68934#discussion_r4256217906
##########
be/src/format/transformer/vorc_transformer.cpp:
##########
@@ -349,6 +355,11 @@ std::unique_ptr<orc::Type> VOrcTransformer::_build_orc_type(
type = orc::createPrimitiveType(orc::DATE);
break;
}
+ case TYPE_TIMESTAMP_NS: {
+ type = orc::createPrimitiveType(orc::TIMESTAMP);
+ type->setAttribute("iceberg.timestamp-unit", "NANOS");
Review Comment:
The TVF/V2 portion is fixed in 19ef150201f4. Schema RPCs now use V2 ORC schema discovery, preserving the NANOS annotation in scalar and nested TIMESTAMP_NS targets before column mapping. Filesystem information honors the RPC range. Local BE tests and an actual S3 TVF round trip pass, including array/struct/map fields and range-boundary values; the TVF schema golden was generated by running the regression suite. Legacy LOAD and format-v1 decoding remain explicitly out of scope, so this thread stays open.
##########
be/src/exec/sink/writer/iceberg/viceberg_partition_writer.cpp:
##########
@@ -84,10 +109,14 @@ Status VIcebergPartitionWriter::open(RuntimeState* state, RuntimeProfile* profil
open_status = parquet_compression_type.error();
break;
}
- ParquetFileOptions parquet_options = {.compression_type = parquet_compression_type.value(),
- .parquet_version = TParquetVersion::PARQUET_1_0,
- .parquet_disable_dictionary = false,
- .enable_int96_timestamps = false};
+ // Parquet 1.0 coerces Arrow nanoseconds to microseconds, including nested fields.
+ ParquetFileOptions parquet_options = {
+ .compression_type = parquet_compression_type.value(),
+ .parquet_version = contains_nanosecond_timestamp(_schema.root_struct())
Review Comment:
The TVF/V2 portion is fixed in 19ef150201f4. Schema RPCs now use V2 Parquet schema discovery, preserving unadjusted TIMESTAMP(NANOS) in scalar and nested TIMESTAMP_NS targets instead of first choosing DATETIMEV2(6). Filesystem information honors the RPC range. Local plain/dictionary BE tests and an actual S3 TVF round trip pass, including array/struct/map fields and range-boundary values; the TVF schema golden was generated by running the regression suite. Legacy LOAD and format-v1 decoding remain explicitly out of scope, so this thread stays open.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](datalake) Support nanosecond timestamps in Iceberg and Paimon [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #68934:
URL: https://github.com/apache/doris/pull/68934#issuecomment-6138190105
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](datalake) Support nanosecond timestamps in Iceberg and Paimon [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #68934:
URL: https://github.com/apache/doris/pull/68934#issuecomment-6182648992
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](datalake) Support nanosecond timestamps in Iceberg and Paimon [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68934:
URL: https://github.com/apache/doris/pull/68934#issuecomment-6140342561
<details>
<summary>TPC-DS: <b>Total hot run time: 152199 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3f7020a33f405beaf37e5adbf07ccbb55a06774a, data reload: false
query5 4317 616 449 449
query6 418 208 197 197
query7 4798 460 229 229
query8 315 178 163 163
query9 8761 3968 4020 3968
query10 460 310 258 258
query11 5997 3413 3186 3186
query12 161 88 87 87
query13 1247 445 335 335
query14 6541 4770 4460 4460
query14_1 4233 4230 4182 4182
query15 221 206 174 174
query16 981 481 444 444
query17 889 696 539 539
query18 2419 428 313 313
query19 202 160 136 136
query20 82 79 82 79
query21 209 133 111 111
query22 12908 13038 12792 12792
query23 13348 12603 12176 12176
query23_1 12356 12334 12268 12268
query24 7076 815 466 466
query24_1 471 474 469 469
query25 534 413 361 361
query26 1264 257 139 139
query27 2756 444 270 270
query28 4623 1951 1956 1951
query29 1554 599 470 470
query30 297 221 176 176
query31 876 764 661 661
query32 141 94 95 94
query33 516 307 252 252
query34 919 846 500 500
query35 757 759 643 643
query36 831 848 752 752
query37 135 104 90 90
query38 1790 1770 1719 1719
query39 729 715 673 673
query39_1 647 675 687 675
query40 227 119 103 103
query41 74 72 68 68
query42 88 84 84 84
query43 367 378 324 324
query44 1301 715 715 715
query45 181 185 160 160
query46 856 967 569 569
query47 3036 3042 2953 2953
query48 290 297 214 214
query49 568 398 290 290
query50 686 279 212 212
query51 10558 10421 10538 10421
query52 74 80 72 72
query53 184 204 152 152
query54 236 183 191 183
query55 81 68 62 62
query56 236 246 229 229
query57 1601 1557 1566 1557
query58 276 256 241 241
query59 2360 2365 2115 2115
query60 274 236 216 216
query61 138 148 145 145
query62 397 336 288 288
query63 188 174 167 167
query64 2737 951 753 753
query65 3417 3341 3369 3341
query66 1781 433 299 299
query67 20273 20231 20019 20019
query68 2883 985 610 610
query69 384 287 247 247
query70 901 832 837 832
query71 297 227 209 209
query72 2699 2634 2213 2213
query73 524 527 299 299
query74 4569 4446 4298 4298
query75 2331 2284 1918 1918
query76 2285 1007 612 612
query77 354 399 300 300
query78 9048 9043 8507 8507
query79 973 900 540 540
query80 726 425 353 353
query81 551 324 277 277
query82 577 134 108 108
query83 271 200 182 182
query84 307 117 104 104
query85 879 430 400 400
query86 332 236 234 234
query87 2029 1997 1862 1862
query88 3619 2742 2687 2687
query89 335 298 259 259
query90 1816 173 176 173
query91 163 151 121 121
query92 94 87 93 87
query93 1012 1010 571 571
query94 475 311 298 298
query95 582 407 316 316
query96 682 527 233 233
query97 2462 2434 2315 2315
query98 154 165 142 142
query99 787 772 660 660
Total cold run time: 234193 ms
Total hot run time: 152199 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](datalake) Support nanosecond timestamps in Iceberg and Paimon [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68934:
URL: https://github.com/apache/doris/pull/68934#issuecomment-6136686026
<details>
<summary>TPC-H: <b>Total hot run time: 28954 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 203abf3a6b955d390f970ace3094d1ddce7bc37a, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17588 5628 5568 5568
q2 2100 303 287 287
q3 10220 1418 829 829
q4 4681 477 359 359
q5 7448 783 525 525
q6 187 189 155 155
q7 759 791 611 611
q8 9241 1372 1104 1104
q9 5803 4532 4539 4532
q10 6821 1309 1017 1017
q11 438 256 243 243
q12 629 405 280 280
q13 18067 3115 2383 2383
q14 272 276 247 247
q15 q16 738 731 665 665
q17 1175 742 572 572
q18 7096 6244 6200 6200
q19 1103 964 599 599
q20 381 347 229 229
q21 5398 2268 2416 2268
q22 357 307 281 281
Total cold run time: 100502 ms
Total hot run time: 28954 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 6173 6142 6169 6142
q2 717 581 545 545
q3 5226 5407 4935 4935
q4 2022 2113 1529 1529
q5 5307 5336 5743 5336
q6 262 207 156 156
q7 2294 2102 1849 1849
q8 3141 2780 2753 2753
q9 8296 8322 8419 8322
q10 3793 3716 3323 3323
q11 647 466 444 444
q12 688 756 516 516
q13 2920 3286 2492 2492
q14 339 305 298 298
q15 q16 727 737 653 653
q17 8506 7507 7469 7469
q18 13218 12353 13155 12353
q19 919 787 779 779
q20 2205 2153 1941 1941
q21 5768 4684 4796 4684
q22 484 456 396 396
Total cold run time: 73652 ms
Total hot run time: 66915 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](datalake) Support nanosecond timestamps in Iceberg and Paimon [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #68934:
URL: https://github.com/apache/doris/pull/68934#issuecomment-6139463089
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](datalake) Support nanosecond timestamps in Iceberg and Paimon [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #68934:
URL: https://github.com/apache/doris/pull/68934#issuecomment-6136204698
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](datalake) Support nanosecond timestamps in Iceberg and Paimon [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #68934:
URL: https://github.com/apache/doris/pull/68934#issuecomment-6118883857
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](datalake) Support nanosecond timestamps in Iceberg and Paimon [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #68934:
URL: https://github.com/apache/doris/pull/68934#discussion_r4245058048
##########
fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonScanPlanProvider.java:
##########
@@ -964,7 +968,7 @@ && hasTrustworthyLimitAccounting(table)
Optional<List<RawFile>> optRawFiles = dataSplit.convertToRawFiles();
Optional<List<DeletionFile>> optDeletionFiles = dataSplit.deletionFiles();
- boolean nativeEligible = shouldUseNativeReader(paimonHandle.isForceJni(),
+ boolean nativeEligible = shouldUseNativeReader(paimonHandle.isForceJni() || needsNanosecondReader,
Review Comment:
[P2] Preserve metadata columns when routing nanosecond scans. A normal Paimon table advertises `__paimon_file_path` and `__paimon_row_index`, but selecting either with a `TIMESTAMP(7-9)` field sets `needsNanosecondReader` and makes `nativeEligible` false here. The JNI branch then calls `validateMetadataColumnReader(true, false)` and rejects every raw-convertible split, whereas its FileScannerV2 native path previously served the query. This needs a reader path that preserves both nanoseconds and physical metadata, plus a regression query covering the combination.
##########
be/src/format_v2/orc/orc_reader.cpp:
##########
@@ -947,6 +953,12 @@ Status OrcReader::init(RuntimeState* state) {
try {
_state->reader = ::orc::createReader(std::move(input_stream), options);
_state->root_type = &_state->reader->getType();
+ if (orc_serde_utils::has_nanosecond_timestamp(*_state->root_type)) {
+ // Use the same UTC wall-clock carrier as the writer; historical local offsets
+ // differ between ORC and cctz and can push boundary values out of range.
+ _state->timezone = "GMT";
Review Comment:
[P1] Keep the ORC negative-fraction correction when changing reader timezone. For an external Iceberg ORC stripe written with `Asia/Shanghai`, a valid `timestamp_ns` wall time `1970-01-01 00:00:00.500000000` enters ORC 1.9.0 as second `-28800`; its writer stores `-28799` for the fractional-second encoding. This new file-wide GMT reader mode shifts that to `+1`, so ORC's reader skips its correction (which runs only for negative adjusted seconds), and Doris returns `00:00:01.500000000`. A plain timestamp in the same file follows the same path. Keep a compatible writer-zone conversion or correct the boundary before materialization, and cover a non-GMT external stripe crossing zero.
##########
fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergPartitionUtils.java:
##########
@@ -479,6 +491,13 @@ static Object parsePartitionValueFromString(String valueStr, Type icebergType, Z
case DATE:
// Iceberg date is days since epoch (1970-01-01).
return (int) LocalDate.parse(valueStr, DateTimeFormatter.ISO_LOCAL_DATE).toEpochDay();
+ case TIMESTAMP_NANO:
+ LocalDateTime timestampNs = LocalDateTime.parse(valueStr.replace('T', ' '),
Review Comment:
[P2] Accept the write-side date-only form for static timestamp_ns partitions. `INSERT OVERWRITE ... PARTITION(ts='2024-01-01')` can materialize a valid midnight `TIMESTAMP_NS` row because its BE cast accepts the ten-character date, but the static partition map retains the raw string and this new `TIMESTAMP_NANO` parser requires `HH:mm:ss`. `buildPartitionFilter` then fails during overwrite commit, after writing files. Normalize the static value to the typed timestamp or parse the accepted date-only form here, and cover it with a static identity overwrite test.
##########
be/src/format/transformer/vorc_transformer.cpp:
##########
@@ -349,6 +355,11 @@ std::unique_ptr<orc::Type> VOrcTransformer::_build_orc_type(
type = orc::createPrimitiveType(orc::DATE);
break;
}
+ case TYPE_TIMESTAMP_NS: {
+ type = orc::createPrimitiveType(orc::TIMESTAMP);
+ type->setAttribute("iceberg.timestamp-unit", "NANOS");
Review Comment:
[P2] Preserve ORC nanoseconds in TVF and LOAD reads. This writer marks the field `iceberg.timestamp-unit=NANOS`, but legacy `OrcReader::convert_to_doris_type` ignores the attribute and infers `DATETIMEV2(6)` for TVF schema fetch and LOAD source columns. With `enable_file_scanner_v2=false`, a TVF scan uses that reader's `nanoseconds / 1000` conversion, returning `.123456789` as `.123456`; LOAD also uses the legacy reader even when V2 is requested. Handle the annotation in inference and decoding, or route both paths through a lossless reader, and test both round trips.
##########
fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonTypeMapping.java:
##########
@@ -156,6 +156,10 @@ private static ConnectorType toDecimalType(DecimalType type) {
private static ConnectorType toTimestampType(DataType dataType) {
int scale = DEFAULT_TIMESTAMP_SCALE;
if (dataType instanceof TimestampType) {
+ // Keep all source fractional digits instead of truncating high-precision wall-clock values.
+ if (((TimestampType) dataType).getPrecision() > MAX_TIMESTAMP_SCALE) {
+ return ConnectorType.of("TIMESTAMP_NS");
Review Comment:
[P2] Preserve access to high-precision Paimon values outside the nanosecond range. This maps every existing Paimon `TimestampType(7-9)` to `TIMESTAMP_NS`, whose signed 64-bit epoch range is only about 1677-2262. A valid existing value such as `1000-01-01 00:00:00` previously mapped to `DATETIMEV2(6)`, but the JNI `appendTimestampNs` path now reaches `Math.multiplyExact` in `TypeNativeBytes.convertToTimestampNs` and aborts the scan. Please retain a compatibility route or make the narrower range opt-in for existing tables, and test an out-of-range historical row.
##########
be/src/exec/sink/writer/iceberg/viceberg_partition_writer.cpp:
##########
@@ -84,10 +109,14 @@ Status VIcebergPartitionWriter::open(RuntimeState* state, RuntimeProfile* profil
open_status = parquet_compression_type.error();
break;
}
- ParquetFileOptions parquet_options = {.compression_type = parquet_compression_type.value(),
- .parquet_version = TParquetVersion::PARQUET_1_0,
- .parquet_disable_dictionary = false,
- .enable_int96_timestamps = false};
+ // Parquet 1.0 coerces Arrow nanoseconds to microseconds, including nested fields.
+ ParquetFileOptions parquet_options = {
+ .compression_type = parquet_compression_type.value(),
+ .parquet_version = contains_nanosecond_timestamp(_schema.root_struct())
Review Comment:
[P2] Preserve Parquet nanoseconds in TVF and LOAD reads. This new Parquet 2 write path emits unadjusted `TIMESTAMP(NANOS)`, but legacy `format/parquet/schema_desc.cpp` declares it `DATETIMEV2(6)`. TVF schema fetch fixes that type before forced FileScannerV2 decodes the physical `TIMESTAMP_NS`, so `ColumnMapper` rounds `.123456789` to `.123457`. LOAD disables V2 and its legacy `Int64ToTimestamp` converter keeps only microseconds. Update both schema inference and legacy load conversion (or route them through lossless decoding), and test both file round trips.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](datalake) Support nanosecond timestamps in Iceberg and Paimon [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #68934:
URL: https://github.com/apache/doris/pull/68934#issuecomment-6118175698
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](datalake) Support nanosecond timestamps in Iceberg and Paimon [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68934:
URL: https://github.com/apache/doris/pull/68934#issuecomment-6127416019
# FE Regression Coverage Report
Increment line coverage ` 75.00% (3/4)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/68934_643eb975c593ce69a4c3592b6a5769e053cb6b6f_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/68934_643eb975c593ce69a4c3592b6a5769e053cb6b6f_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](datalake) Support nanosecond timestamps in Iceberg and Paimon [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #68934:
URL: https://github.com/apache/doris/pull/68934#issuecomment-6137615255
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](datalake) Support nanosecond timestamps in Iceberg and Paimon [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68934:
URL: https://github.com/apache/doris/pull/68934#issuecomment-6138079806
<details>
<summary>ClickBench: <b>Total hot run time: 24.64 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a48a522100127108c22a0319a9a82a7781e0b4c4, data reload: false
query1 0.01 0.01 0.01
query2 0.09 0.05 0.05
query3 0.27 0.12 0.12
query4 1.65 0.12 0.12
query5 0.25 0.25 0.25
query6 1.18 0.68 0.68
query7 0.04 0.01 0.01
query8 0.07 0.04 0.04
query9 0.44 0.36 0.37
query10 0.58 0.58 0.57
query11 0.21 0.15 0.13
query12 0.20 0.14 0.15
query13 0.49 0.50 0.51
query14 0.98 0.99 0.96
query15 0.65 0.62 0.60
query16 0.35 0.35 0.35
query17 1.12 1.18 1.13
query18 0.23 0.21 0.21
query19 2.16 2.07 2.08
query20 0.01 0.01 0.02
query21 15.50 0.26 0.14
query22 4.45 0.05 0.06
query23 16.30 0.33 0.12
query24 3.20 0.63 0.40
query25 0.11 0.07 0.05
query26 1.08 0.26 0.16
query27 0.05 0.04 0.03
query28 1.72 0.65 0.40
query29 12.51 4.22 3.29
query30 0.28 0.14 0.13
query31 2.78 0.60 0.34
query32 3.22 0.62 0.52
query33 3.25 3.20 3.19
query34 15.43 4.04 3.38
query35 3.38 3.34 3.36
query36 0.55 0.44 0.45
query37 0.10 0.06 0.06
query38 0.05 0.03 0.04
query39 0.04 0.03 0.03
query40 0.17 0.15 0.14
query41 0.09 0.03 0.03
query42 0.04 0.03 0.03
query43 0.04 0.04 0.03
Total cold run time: 95.32 s
Total hot run time: 24.64 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](datalake) Support nanosecond timestamps in Iceberg and Paimon [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68934:
URL: https://github.com/apache/doris/pull/68934#issuecomment-6140200568
<details>
<summary>TPC-H: <b>Total hot run time: 29066 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3f7020a33f405beaf37e5adbf07ccbb55a06774a, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17590 5653 5524 5524
q2 2098 323 260 260
q3 10233 1420 843 843
q4 4712 468 352 352
q5 7432 784 541 541
q6 183 186 156 156
q7 758 814 616 616
q8 9266 1395 1077 1077
q9 5751 4470 4429 4429
q10 6807 1292 1001 1001
q11 432 261 244 244
q12 632 400 286 286
q13 18062 3079 2411 2411
q14 264 274 246 246
q15 q16 745 736 662 662
q17 1211 758 581 581
q18 7094 6231 6213 6213
q19 1109 962 590 590
q20 382 335 220 220
q21 5993 2525 2634 2525
q22 393 358 289 289
Total cold run time: 101147 ms
Total hot run time: 29066 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 6636 6653 6837 6653
q2 731 602 571 571
q3 5599 5767 5272 5272
q4 2047 2140 1586 1586
q5 5567 5725 5748 5725
q6 255 196 147 147
q7 2220 2017 1769 1769
q8 3177 2735 2688 2688
q9 8263 8223 7982 7982
q10 3673 3612 3237 3237
q11 597 405 367 367
q12 670 689 497 497
q13 2753 3058 2382 2382
q14 280 295 265 265
q15 q16 678 691 605 605
q17 7858 7088 6897 6897
q18 13078 12261 13029 12261
q19 927 804 794 794
q20 2158 2172 1962 1962
q21 5807 4686 4783 4686
q22 496 461 388 388
Total cold run time: 73470 ms
Total hot run time: 66734 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](datalake) Support nanosecond timestamps in Iceberg and Paimon [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68934:
URL: https://github.com/apache/doris/pull/68934#issuecomment-6183674448
<details>
<summary>TPC-DS: <b>Total hot run time: 152276 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 85d1412fccf4c085604a751c48e00dcce30e71b7, data reload: false
query5 4326 618 473 473
query6 424 198 189 189
query7 4804 439 236 236
query8 330 170 154 154
query9 8733 3992 3992 3992
query10 471 307 255 255
query11 6052 3813 3519 3519
query12 152 93 91 91
query13 1267 467 306 306
query14 6084 4010 3781 3781
query14_1 3535 3516 3497 3497
query15 206 195 182 182
query16 957 441 413 413
query17 862 657 554 554
query18 2432 437 315 315
query19 205 170 138 138
query20 85 86 87 86
query21 214 133 114 114
query22 12975 12954 12803 12803
query23 13538 13073 12568 12568
query23_1 12675 12538 12643 12538
query24 7093 847 473 473
query24_1 474 476 489 476
query25 538 424 372 372
query26 1277 256 149 149
query27 2762 445 281 281
query28 4604 1996 1949 1949
query29 1569 610 457 457
query30 300 227 189 189
query31 1003 926 817 817
query32 146 99 95 95
query33 529 308 245 245
query34 902 846 506 506
query35 767 779 645 645
query36 844 807 730 730
query37 136 111 102 102
query38 1809 1760 1699 1699
query39 804 804 756 756
query39_1 741 743 745 743
query40 227 118 102 102
query41 78 69 72 69
query42 90 83 84 83
query43 375 388 334 334
query44 1344 706 715 706
query45 185 187 170 170
query46 869 965 574 574
query47 2944 3008 2976 2976
query48 306 311 213 213
query49 571 393 293 293
query50 690 286 213 213
query51 10615 10602 10346 10346
query52 76 87 69 69
query53 191 220 157 157
query54 252 192 190 190
query55 70 65 68 65
query56 243 229 211 211
query57 1649 1569 1538 1538
query58 289 263 252 252
query59 2087 2121 1942 1942
query60 291 245 230 230
query61 151 153 150 150
query62 399 336 287 287
query63 193 169 163 163
query64 2724 951 809 809
query65 3293 3221 3251 3221
query66 1783 408 293 293
query67 20152 20048 19747 19747
query68 3037 1051 623 623
query69 388 283 232 232
query70 872 779 751 751
query71 295 227 212 212
query72 2773 2305 2295 2295
query73 532 525 291 291
query74 4602 4445 4311 4311
query75 2334 2282 1947 1947
query76 2300 1016 607 607
query77 349 377 276 276
query78 9871 9739 9216 9216
query79 981 896 516 516
query80 790 433 358 358
query81 557 310 276 276
query82 586 138 108 108
query83 284 205 202 202
query84 315 118 102 102
query85 966 428 381 381
query86 351 248 227 227
query87 1989 1941 1822 1822
query88 3631 2710 2691 2691
query89 335 292 256 256
query90 1707 184 185 184
query91 162 148 125 125
query92 100 89 93 89
query93 1030 1043 591 591
query94 492 305 272 272
query95 594 341 375 341
query96 675 518 234 234
query97 2473 2437 2334 2334
query98 159 151 151 151
query99 794 765 639 639
Total cold run time: 234790 ms
Total hot run time: 152276 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](datalake) Support nanosecond timestamps in Iceberg and Paimon [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #68934:
URL: https://github.com/apache/doris/pull/68934#issuecomment-6119754428
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](datalake) Support nanosecond timestamps in Iceberg and Paimon [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #68934:
URL: https://github.com/apache/doris/pull/68934#issuecomment-6118442075
Codex automated review failed and did not complete.
Error: Review recovery stopped: No final review submission was declared
Workflow run: https://github.com/apache/doris/actions/runs/38206980703
Please inspect the workflow logs and rerun the review after the underlying issue is resolved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](datalake) Support nanosecond timestamps in Iceberg and Paimon [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68934:
URL: https://github.com/apache/doris/pull/68934#issuecomment-6183560946
<details>
<summary>TPC-H: <b>Total hot run time: 29542 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 85d1412fccf4c085604a751c48e00dcce30e71b7, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17609 5685 5547 5547
q2 2110 317 264 264
q3 10236 1414 862 862
q4 4690 478 347 347
q5 7441 799 546 546
q6 190 187 152 152
q7 767 797 618 618
q8 9264 1369 1109 1109
q9 5546 4293 4278 4278
q10 6815 1299 1031 1031
q11 445 259 237 237
q12 634 401 289 289
q13 18072 3598 2966 2966
q14 273 275 249 249
q15 q16 745 723 655 655
q17 1201 744 574 574
q18 7076 6252 6184 6184
q19 1097 953 601 601
q20 386 333 229 229
q21 5408 2532 2640 2532
q22 395 355 272 272
Total cold run time: 100400 ms
Total hot run time: 29542 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 6520 6516 6421 6421
q2 760 627 573 573
q3 5559 5745 5298 5298
q4 2320 2169 1587 1587
q5 5760 5636 5749 5636
q6 257 203 150 150
q7 2373 1987 1813 1813
q8 3172 2812 2753 2753
q9 8250 7700 7611 7611
q10 3690 3617 3251 3251
q11 626 412 381 381
q12 678 690 498 498
q13 3291 3666 3000 3000
q14 279 306 265 265
q15 q16 668 690 629 629
q17 7909 6958 6855 6855
q18 13229 12235 13188 12235
q19 940 820 816 816
q20 2194 2185 1940 1940
q21 5853 4720 4812 4720
q22 494 454 420 420
Total cold run time: 74822 ms
Total hot run time: 66852 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](datalake) Support nanosecond timestamps in Iceberg and Paimon [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68934:
URL: https://github.com/apache/doris/pull/68934#issuecomment-6185572750
# FE Regression Coverage Report
Increment line coverage ` 75.00% (3/4)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/68934_85d1412fccf4c085604a751c48e00dcce30e71b7_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/68934_85d1412fccf4c085604a751c48e00dcce30e71b7_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](datalake) Support nanosecond timestamps in Iceberg and Paimon [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #68934:
URL: https://github.com/apache/doris/pull/68934#discussion_r4245321064
##########
fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergTypeMapping.java:
##########
@@ -150,12 +150,16 @@ private static ConnectorType fromPrimitive(Type.PrimitiveType primitive,
return ConnectorType.of("TIMESTAMPTZ", ICEBERG_DATETIME_SCALE_MS, 0);
}
return ConnectorType.of("DATETIMEV2", ICEBERG_DATETIME_SCALE_MS, 0);
+ case TIMESTAMP_NANO:
+ // TIMESTAMP_NS is a wall-clock type; zoned nanoseconds need a distinct carrier.
+ return ConnectorType.of(((Types.TimestampNanoType) primitive).shouldAdjustToUTC()
+ ? "UNSUPPORTED" : "TIMESTAMP_NS");
Review Comment:
[P2] Convert timestamp-nano literals for filtered Iceberg rewrites. This new mapping makes v3 `timestamp_ns` columns queryable, but `IcebergPredicateConverter.extractIcebergLiteral` still handles only `TypeID.TIMESTAMP` for datetime/string literals. A `rewrite_data_files` call with `WHERE ts_ns = '2024-01-02 03:04:05.123456789'` therefore drops its sole predicate, and `RewriteDataFilePlanner` rejects the operation as unpushable. Convert these literals to exact epoch nanoseconds for `TIMESTAMP_NANO` and cover a filtered rewrite; normal scans also currently lose file pruning.
##########
fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergTypeMapping.java:
##########
@@ -218,6 +222,8 @@ static Type toIcebergPrimitive(ConnectorType type) {
// Legacy parity: timestamp WITHOUT zone (datetime scale dropped — Iceberg timestamps are
// microsecond). DorisTypeToIcebergType.atomic returns TimestampType.withoutZone().
return Types.TimestampType.withoutZone();
+ case "TIMESTAMP_NS":
Review Comment:
[P2] Format timestamp-nano write defaults before Iceberg writes. This mapping makes an unzoned v3 `timestamp_ns` field writable, but `IcebergWriteSchemaContext` formats every non-null `writeDefault()` while building the sink schema and its `toDorisSql` switch has no `TIMESTAMP_NANO` arm. An INSERT into a table with such a default throws `Unsupported Iceberg write-default type` before planning, even if the INSERT supplies that column. Add a nanosecond-preserving SQL literal conversion and cover both explicit and omitted-column INSERTs.
##########
fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergTypeMapping.java:
##########
@@ -218,6 +222,8 @@ static Type toIcebergPrimitive(ConnectorType type) {
// Legacy parity: timestamp WITHOUT zone (datetime scale dropped — Iceberg timestamps are
// microsecond). DorisTypeToIcebergType.atomic returns TimestampType.withoutZone().
return Types.TimestampType.withoutZone();
+ case "TIMESTAMP_NS":
+ return Types.TimestampNanoType.withoutZone();
Review Comment:
[P2] Parse timestamp-nano defaults accepted by ADD COLUMN. This new `TIMESTAMP_NS` mapping lets a v3 Iceberg table add a nanosecond timestamp column, and FE validation accepts `DEFAULT '2024-01-02 03:04:05.123456789'`. `IcebergConnectorMetadata.toAddColumnChange` then passes the unchanged text to `IcebergSchemaBuilder.parseDefaultLiteral`, whose `TIMESTAMP_NANO` arm calls `Long.parseLong` and rejects the ALTER before updating the schema. Convert the validated timestamp to exact epoch nanoseconds here and cover ADD COLUMN with a nine-digit default. This is separate from formatting an existing write default during INSERT.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](datalake) Support nanosecond timestamps in Iceberg and Paimon [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #68934:
URL: https://github.com/apache/doris/pull/68934#discussion_r4247525820
##########
fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergPredicateConverter.java:
##########
@@ -671,6 +679,19 @@ private long toMicros(LocalDateTime dateTime, Type icebergType, boolean utcCarri
return instant.getEpochSecond() * 1_000_000L + instant.getNano() / 1000L;
}
+ private static Literal<Long> toNanos(LocalDateTime dateTime, Type icebergType) {
+ if (((Types.TimestampNanoType) icebergType).shouldAdjustToUTC()) {
+ return null;
+ }
+ try {
+ // A raw Long is a microsecond literal in Iceberg; preserve the nanosecond unit explicitly.
+ // Unzoned values use a UTC wall-clock carrier, never the session zone.
+ return Expressions.nanos(IcebergPartitionUtils.toEpochNanos(dateTime));
Review Comment:
[P1] Preserve historical DATE bounds before nanos pushdown. After a valid v3 DATE-to-`timestamp_ns` promotion, a retained DATE file containing `2024-01-02` still has four-byte manifest bounds equal to epoch-day 19724. This branch pushes nanos equality at 1704153600000000000, but the [pinned Iceberg 1.11 conversion](https://github.com/apache/iceberg/blob/apache-iceberg-1.11.0/api/src/main/java/org/apache/iceberg/types/Conversions.java#L156-L164) widens those bounds to the unscaled long 19724. `InclusiveMetricsEvaluator` therefore prunes the matching file: the equality query returns no rows while an unfiltered native scan correctly returns midnight. Both SDK and cached planning use this evaluator. Interpret retained DATE bounds in the promoted domain or conservatively suppress this pushdown, and test old files with real bounds; strict CAST checks cannot repair file pruning.
##########
fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergTypeMapping.java:
##########
@@ -150,12 +150,16 @@ private static ConnectorType fromPrimitive(Type.PrimitiveType primitive,
return ConnectorType.of("TIMESTAMPTZ", ICEBERG_DATETIME_SCALE_MS, 0);
}
return ConnectorType.of("DATETIMEV2", ICEBERG_DATETIME_SCALE_MS, 0);
+ case TIMESTAMP_NANO:
Review Comment:
[P2] Fence nanosecond scans from older backend readers. During a cloud smooth upgrade, a live source BE from the PR base remains query eligible and already supports the internal `TIMESTAMP_NS` carrier. However, its V2 Parquet reader infers unadjusted NANOS as `DATETIMEV2(6)`, truncates the fraction, then casts that result to the new FE's `TIMESTAMP_NS` target: `.123456789` becomes `.123456000`. A plain optional unpartitioned nanos projection does not trigger `requiresCurrentScanSemantics`, so `PluginDrivenScanNode` skips its smooth-source guard and scheduling admits that BE. Base and head also share execution version 16, so the existing internal-type gate cannot distinguish their file semantics. Enforce the required reader capability/version for scalar, nested and filter-only nanos fields, and cover an old source backend in planning tests.
##########
fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergTypeMapping.java:
##########
@@ -150,12 +150,16 @@ private static ConnectorType fromPrimitive(Type.PrimitiveType primitive,
return ConnectorType.of("TIMESTAMPTZ", ICEBERG_DATETIME_SCALE_MS, 0);
}
return ConnectorType.of("DATETIMEV2", ICEBERG_DATETIME_SCALE_MS, 0);
+ case TIMESTAMP_NANO:
+ // TIMESTAMP_NS is a wall-clock type; zoned nanoseconds need a distinct carrier.
+ return ConnectorType.of(((Types.TimestampNanoType) primitive).shouldAdjustToUTC()
+ ? "UNSUPPORTED" : "TIMESTAMP_NS");
Review Comment:
[P2] Fail on out-of-range historical DATE promotion. For an unpartitioned v3 table that promotes an optional DATE field to `timestamp_ns` while retaining a Parquet DATE file containing `1000-01-01`, this mapping now exposes the column as `TIMESTAMP_NS`. The field-ID mapper builds ordinary `CAST(DATEV2 AS TIMESTAMP_NS)`; with default `enable_strict_cast=false`, its overflow becomes NULL and `TableReader` successfully returns that NULL. The [Iceberg schema-evolution contract](https://iceberg.apache.org/spec/#schema-evolution) requires a runtime failure outside the promoted range. This metadata is accepted by the pinned parser and can be published through `TableMetadata.Builder.setCurrentSchema`/`TableOperations.commit`, despite the narrower `UpdateSchema` convenience API. Use checked promotion or explicitly reject the unsupported conversion, and cover retained DATE files with an out-of-range value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](datalake) Support nanosecond timestamps in Iceberg and Paimon [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #68934:
URL: https://github.com/apache/doris/pull/68934#issuecomment-6136997027
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](datalake) Support nanosecond timestamps in Iceberg and Paimon [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #68934:
URL: https://github.com/apache/doris/pull/68934#issuecomment-6136997509
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](datalake) Support nanosecond timestamps in Iceberg and Paimon [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #68934:
URL: https://github.com/apache/doris/pull/68934#discussion_r4256804111
##########
fe/be-java-extensions/paimon-scanner/src/main/java/org/apache/doris/paimon/PaimonArrowBatchAdapter.java:
##########
@@ -163,9 +164,12 @@ public Timestamp getTimestamp(int index, int precision) {
case MICROSECOND:
return Timestamp.fromMicros(value);
case NANOSECOND:
- return Timestamp.fromEpochMillis(
- Math.floorDiv(value, 1_000_000L),
- (int) Math.floorMod(value, 1_000_000L));
+ // Paimon serializers trust target precision. Truncate the positive
+ // fractional remainder so negative epochs and leading zero digits
+ // obey the same narrowing rule, including nested values and keys.
+ int nanos = (int) Math.floorMod(value, 1_000_000L);
+ return Timestamp.fromEpochMillis(Math.floorDiv(value, 1_000_000L),
+ nanos / nanosPerUnit * nanosPerUnit);
Review Comment:
[P2] Reject narrowing that leaves the readable nanos range. For an existing Paimon `TIMESTAMP(7/8)`, Doris exposes the sink column as `TIMESTAMP_NS` and accepts `1677-09-21 00:12:43.145224192` (`Long.MIN_VALUE`). This branch truncates it to `.145224100`/`.145224190`, respectively 92ns/2ns below the signed-nanos minimum. [Paimon's serializer](https://github.com/apache/paimon/blob/release-1.4.2/paimon-common/src/main/java/org/apache/paimon/data/serializer/TimestampSerializer.java#L49-L57) can persist both millisecond-plus-tail values, and `tableRow`/`writer.write` adds no signed-nanos check, so the insert can commit; subsequent Doris JNI scans overflow in `TypeNativeBytes.convertToTimestampNs`. Check that the normalized result remains in the readable range and reject before `writer.write`, including nested fields, with precision-7/8 lower-bound write/read coverage. This differs from the existing historical-row range thread: a supported in-range Doris input creates the unreadable val
ue, and rejecting that write requires no mapping fallback.
##########
be/src/format_v2/parquet/parquet_scan.cpp:
##########
@@ -1579,6 +1579,7 @@ bool get_typed_dictionary_raw_values(PrimitiveType primitive_type, const IColumn
GET_TYPED_DICTIONARY_VALUES(TYPE_DATEV2);
GET_TYPED_DICTIONARY_VALUES(TYPE_DATETIMEV2);
GET_TYPED_DICTIONARY_VALUES(TYPE_TIMESTAMPTZ);
+ GET_TYPED_DICTIONARY_VALUES(TYPE_TIMESTAMP_NS);
Review Comment:
[P2] Preserve fused dictionary projection for nanos columns. For a flat, fully dictionary-encoded NANOS column, `SELECT ts FROM t WHERE ts >= CAST('2024-01-02 03:04:05.123456789' AS TIMESTAMP_NS)` now reaches dictionary-ID filtering with `ColumnTimeStampNs`. `try_filter_and_project_fixed_width_dictionary` still omits that class, so each survivor goes into a temporary ID column, is copied and revalidated by `append_dictionary_values`, and is gathered in a separate pass. `try_simd_insert_parquet_dictionary_indices` also omits `TYPE_TIMESTAMP_NS`, forcing the scalar gather. The previous `ColumnDateTimeV2` carrier used the fused filter/value gather, so normal dictionary scans incur extra passes and buffer traffic. Register the nanos type in both dispatchers and add a projected nanos dictionary-predicate test checking exact values and `DictionaryPredicateFusedProjectedRows`. This is separate from the repaired nullable-selection thread: it occurs without NULLs in dictionary predicate pr
ojection.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](datalake) Support nanosecond timestamps in Iceberg and Paimon [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on code in PR #68934:
URL: https://github.com/apache/doris/pull/68934#discussion_r4255723965
##########
be/src/format/transformer/vorc_transformer.cpp:
##########
@@ -349,6 +355,11 @@ std::unique_ptr<orc::Type> VOrcTransformer::_build_orc_type(
type = orc::createPrimitiveType(orc::DATE);
break;
}
+ case TYPE_TIMESTAMP_NS: {
+ type = orc::createPrimitiveType(orc::TIMESTAMP);
+ type->setAttribute("iceberg.timestamp-unit", "NANOS");
Review Comment:
I will fix ORC TVF schema discovery and the format-v2 read path so a NANOS annotation reaches the TIMESTAMP_NS target without an intermediate microsecond column. The legacy LOAD / format-v1 decoding path is explicitly out of scope for this PR; I will keep this thread open to make that remaining part visible.
##########
be/src/exec/sink/writer/iceberg/viceberg_partition_writer.cpp:
##########
@@ -84,10 +109,14 @@ Status VIcebergPartitionWriter::open(RuntimeState* state, RuntimeProfile* profil
open_status = parquet_compression_type.error();
break;
}
- ParquetFileOptions parquet_options = {.compression_type = parquet_compression_type.value(),
- .parquet_version = TParquetVersion::PARQUET_1_0,
- .parquet_disable_dictionary = false,
- .enable_int96_timestamps = false};
+ // Parquet 1.0 coerces Arrow nanoseconds to microseconds, including nested fields.
+ ParquetFileOptions parquet_options = {
+ .compression_type = parquet_compression_type.value(),
+ .parquet_version = contains_nanosecond_timestamp(_schema.root_struct())
Review Comment:
I will fix Parquet TVF schema discovery and the format-v2 read path so unadjusted NANOS maps to TIMESTAMP_NS without intermediate microsecond rounding. Legacy LOAD / format-v1 decoding is explicitly out of scope; I will keep this thread open for that remaining part.
##########
fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonTypeMapping.java:
##########
@@ -156,6 +156,10 @@ private static ConnectorType toDecimalType(DecimalType type) {
private static ConnectorType toTimestampType(DataType dataType) {
int scale = DEFAULT_TIMESTAMP_SCALE;
if (dataType instanceof TimestampType) {
+ // Keep all source fractional digits instead of truncating high-precision wall-clock values.
+ if (((TimestampType) dataType).getPrecision() > MAX_TIMESTAMP_SCALE) {
+ return ConnectorType.of("TIMESTAMP_NS");
Review Comment:
This range-compatibility issue is intentionally out of scope, as agreed with the author. High-precision external timestamps will retain the TIMESTAMP_NS mapping; this change will not introduce an opt-in mapping switch or a fallback to DATETIMEV2 for out-of-range values. I will leave the thread open rather than mark it fixed.
##########
fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergTypeMapping.java:
##########
@@ -150,12 +150,16 @@ private static ConnectorType fromPrimitive(Type.PrimitiveType primitive,
return ConnectorType.of("TIMESTAMPTZ", ICEBERG_DATETIME_SCALE_MS, 0);
}
return ConnectorType.of("DATETIMEV2", ICEBERG_DATETIME_SCALE_MS, 0);
+ case TIMESTAMP_NANO:
Review Comment:
Mixed-version FE/BE compatibility is intentionally out of scope for this PR, as agreed with the author. I will not add an older-backend fallback or compatibility gate, and will leave this thread open rather than mark it fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](datalake) Support nanosecond timestamps in Iceberg and Paimon [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68934:
URL: https://github.com/apache/doris/pull/68934#issuecomment-6124967188
<details>
<summary>TPC-H: <b>Total hot run time: 29339 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 643eb975c593ce69a4c3592b6a5769e053cb6b6f, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17617 5731 5595 5595
q2 2091 315 253 253
q3 10239 1421 858 858
q4 4691 480 352 352
q5 7448 814 551 551
q6 203 195 162 162
q7 789 818 613 613
q8 9247 1493 1251 1251
q9 5872 4547 4536 4536
q10 6819 1302 1028 1028
q11 436 269 238 238
q12 641 418 299 299
q13 18058 3137 2404 2404
q14 281 273 252 252
q15 q16 737 729 665 665
q17 1198 764 596 596
q18 7044 6242 6185 6185
q19 1116 973 634 634
q20 389 346 237 237
q21 4830 2358 2519 2358
q22 366 317 272 272
Total cold run time: 100112 ms
Total hot run time: 29339 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 6228 6190 6202 6190
q2 712 580 548 548
q3 5256 5453 4940 4940
q4 2153 2229 1643 1643
q5 5289 5550 5730 5550
q6 272 207 165 165
q7 2361 2050 1938 1938
q8 3267 2836 2958 2836
q9 8270 8291 8565 8291
q10 3781 3817 3395 3395
q11 677 502 452 452
q12 764 775 543 543
q13 2944 3280 2567 2567
q14 320 334 293 293
q15 q16 773 772 667 667
q17 8584 7751 7355 7355
q18 13239 12307 13189 12307
q19 1023 872 864 864
q20 2217 2179 1938 1938
q21 6276 5061 5122 5061
q22 534 452 406 406
Total cold run time: 74940 ms
Total hot run time: 67949 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](datalake) Support nanosecond timestamps in Iceberg and Paimon [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #68934:
URL: https://github.com/apache/doris/pull/68934#issuecomment-6182647057
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](datalake) Support nanosecond timestamps in Iceberg and Paimon [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #68934:
URL: https://github.com/apache/doris/pull/68934#issuecomment-6141209618
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](datalake) Support nanosecond timestamps in Iceberg and Paimon [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68934:
URL: https://github.com/apache/doris/pull/68934#issuecomment-6144523423
<details>
<summary>TPC-H: <b>Total hot run time: 29347 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 19ef150201f4c4928c97b1efbe6003db6b83caf3, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17606 5647 5540 5540
q2 2109 314 266 266
q3 10253 1422 850 850
q4 4684 481 366 366
q5 7443 803 558 558
q6 192 190 157 157
q7 792 815 633 633
q8 9274 1351 1156 1156
q9 5936 4472 4496 4472
q10 6823 1309 1032 1032
q11 428 250 235 235
q12 630 402 288 288
q13 18064 3056 2409 2409
q14 285 263 250 250
q15 q16 731 737 695 695
q17 1194 754 569 569
q18 7193 6300 6226 6226
q19 1121 982 615 615
q20 398 334 225 225
q21 5952 2498 2679 2498
q22 405 351 307 307
Total cold run time: 101513 ms
Total hot run time: 29347 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 6662 6739 6872 6739
q2 721 610 569 569
q3 5644 5831 5380 5380
q4 2079 2160 1588 1588
q5 5697 5515 5758 5515
q6 242 194 146 146
q7 2267 2036 1803 1803
q8 3258 2727 2756 2727
q9 8246 8240 8030 8030
q10 3668 3603 3259 3259
q11 590 428 376 376
q12 667 705 512 512
q13 2744 3107 2390 2390
q14 280 287 261 261
q15 q16 671 675 615 615
q17 7765 7035 6890 6890
q18 13139 12283 13235 12283
q19 913 784 787 784
q20 2175 2169 1949 1949
q21 5880 4752 4809 4752
q22 503 445 417 417
Total cold run time: 73811 ms
Total hot run time: 66985 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](datalake) Support nanosecond timestamps in Iceberg and Paimon [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68934:
URL: https://github.com/apache/doris/pull/68934#issuecomment-6136798106
<details>
<summary>TPC-DS: <b>Total hot run time: 151685 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 203abf3a6b955d390f970ace3094d1ddce7bc37a, data reload: false
query5 4304 589 447 447
query6 415 203 195 195
query7 4814 442 230 230
query8 336 182 163 163
query9 8760 3958 3966 3958
query10 468 315 253 253
query11 5929 3420 3180 3180
query12 145 87 82 82
query13 1233 432 322 322
query14 6524 4810 4499 4499
query14_1 4267 4312 4174 4174
query15 208 209 185 185
query16 952 443 422 422
query17 857 663 548 548
query18 2409 423 320 320
query19 200 165 148 148
query20 85 91 79 79
query21 216 133 110 110
query22 13029 12985 12762 12762
query23 13122 12562 12139 12139
query23_1 12223 12329 12235 12235
query24 7118 833 476 476
query24_1 483 482 483 482
query25 533 422 365 365
query26 1263 260 143 143
query27 2744 437 273 273
query28 4634 1977 1985 1977
query29 1598 560 439 439
query30 284 212 183 183
query31 855 752 637 637
query32 148 104 80 80
query33 508 295 229 229
query34 957 841 499 499
query35 742 748 642 642
query36 819 809 701 701
query37 125 96 85 85
query38 1790 1763 1699 1699
query39 700 707 677 677
query39_1 654 655 661 655
query40 234 120 94 94
query41 65 64 66 64
query42 82 83 80 80
query43 366 381 327 327
query44 1296 711 701 701
query45 172 170 163 163
query46 865 967 579 579
query47 2960 3045 2851 2851
query48 288 300 218 218
query49 564 395 277 277
query50 684 272 213 213
query51 10260 10458 10136 10136
query52 78 79 74 74
query53 186 206 151 151
query54 242 188 191 188
query55 71 70 70 70
query56 209 215 202 202
query57 1631 1640 1554 1554
query58 272 258 243 243
query59 2349 2381 2122 2122
query60 263 232 221 221
query61 143 145 140 140
query62 382 344 287 287
query63 192 178 161 161
query64 2741 932 827 827
query65 3433 3389 3394 3389
query66 1766 411 292 292
query67 20188 20234 19980 19980
query68 2950 1000 622 622
query69 380 286 233 233
query70 933 869 814 814
query71 283 223 210 210
query72 2673 2560 2307 2307
query73 530 532 299 299
query74 4559 4441 4299 4299
query75 2330 2305 1933 1933
query76 2300 1005 607 607
query77 353 394 302 302
query78 8934 9026 8540 8540
query79 873 865 524 524
query80 473 417 358 358
query81 518 328 270 270
query82 209 133 109 109
query83 187 195 178 178
query84 286 120 96 96
query85 762 412 366 366
query86 281 234 233 233
query87 2024 1952 1832 1832
query88 3643 2736 2707 2707
query89 358 298 253 253
query90 2099 181 178 178
query91 178 151 121 121
query92 99 84 87 84
query93 980 1011 581 581
query94 442 326 272 272
query95 582 399 318 318
query96 663 532 240 240
query97 2457 2428 2329 2329
query98 157 146 139 139
query99 767 763 656 656
Total cold run time: 232583 ms
Total hot run time: 151685 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](datalake) Support nanosecond timestamps in Iceberg and Paimon [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #68934:
URL: https://github.com/apache/doris/pull/68934#issuecomment-6120287670
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](datalake) Support nanosecond timestamps in Iceberg and Paimon [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68934:
URL: https://github.com/apache/doris/pull/68934#issuecomment-6121412792
<details>
<summary>ClickBench: <b>Total hot run time: 24.56 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5024d4e894591b18ef797542542aa1b040d73696, data reload: false
query1 0.01 0.01 0.00
query2 0.10 0.05 0.05
query3 0.26 0.13 0.12
query4 1.61 0.13 0.12
query5 0.27 0.25 0.26
query6 1.17 0.69 0.68
query7 0.04 0.00 0.00
query8 0.05 0.04 0.04
query9 0.42 0.35 0.35
query10 0.60 0.59 0.57
query11 0.21 0.15 0.14
query12 0.19 0.14 0.14
query13 0.48 0.48 0.49
query14 0.97 0.98 0.96
query15 0.64 0.61 0.60
query16 0.34 0.36 0.34
query17 1.16 1.17 1.15
query18 0.23 0.21 0.21
query19 2.11 2.01 1.96
query20 0.02 0.01 0.02
query21 15.48 0.28 0.14
query22 4.29 0.06 0.05
query23 16.26 0.32 0.12
query24 3.24 0.54 0.45
query25 0.12 0.06 0.04
query26 1.12 0.25 0.15
query27 0.05 0.03 0.04
query28 1.95 0.63 0.40
query29 12.50 4.27 3.32
query30 0.28 0.14 0.14
query31 2.78 0.58 0.33
query32 3.23 0.62 0.52
query33 3.29 3.24 3.20
query34 15.64 4.04 3.38
query35 3.33 3.36 3.36
query36 0.57 0.42 0.45
query37 0.10 0.06 0.06
query38 0.04 0.04 0.03
query39 0.04 0.03 0.03
query40 0.18 0.15 0.13
query41 0.08 0.03 0.03
query42 0.04 0.03 0.03
query43 0.04 0.03 0.03
Total cold run time: 95.53 s
Total hot run time: 24.56 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](datalake) Support nanosecond timestamps in Iceberg and Paimon [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #68934:
URL: https://github.com/apache/doris/pull/68934#issuecomment-6118348587
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](datalake) Support nanosecond timestamps in Iceberg and Paimon [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #68934:
URL: https://github.com/apache/doris/pull/68934#issuecomment-6141211985
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](datalake) Support nanosecond timestamps in Iceberg and Paimon [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #68934:
URL: https://github.com/apache/doris/pull/68934#discussion_r4278367745
##########
fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonTypeMapping.java:
##########
@@ -156,6 +156,10 @@ private static ConnectorType toDecimalType(DecimalType type) {
private static ConnectorType toTimestampType(DataType dataType) {
int scale = DEFAULT_TIMESTAMP_SCALE;
if (dataType instanceof TimestampType) {
+ // Keep all source fractional digits instead of truncating high-precision wall-clock values.
+ if (((TimestampType) dataType).getPrecision() > MAX_TIMESTAMP_SCALE) {
+ return ConnectorType.of("TIMESTAMP_NS");
Review Comment:
[P2] Keep ORC OUTFILE compatible with the new timestamp mapping
For an existing Paimon `TIMESTAMP(7/8/9)` column, `SELECT ts FROM t INTO OUTFILE ... FORMAT AS ORC` now fails during planning: `PhysicalPlanTranslator.visitPhysicalFileSink` calls `OutFileClause.analyzeForOrcFormat`, whose schema generator and validation switch omit `TIMESTAMP_NS` and throw `currently orc do not support column type: TIMESTAMP_NS`. The previous `DATETIMEV2(6)` mapping exported successfully, so ordinary in-range values regress too. Wire the new type through OUTFILE schema generation/validation and the explicit-schema writer path, and add scalar/nested export/readback coverage; OUTFILE supplies `orc_schema` and bypasses the new `_build_orc_type` branch.
##########
fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonTypeMapping.java:
##########
@@ -156,6 +156,10 @@ private static ConnectorType toDecimalType(DecimalType type) {
private static ConnectorType toTimestampType(DataType dataType) {
int scale = DEFAULT_TIMESTAMP_SCALE;
if (dataType instanceof TimestampType) {
+ // Keep all source fractional digits instead of truncating high-precision wall-clock values.
+ if (((TimestampType) dataType).getPrecision() > MAX_TIMESTAMP_SCALE) {
Review Comment:
[P2] Select nanos-compatible Parquet options for generic exports
For an existing Paimon `TIMESTAMP(9)` value such as `2024-01-02 03:04:05.123456789`, `SELECT ts FROM t INTO OUTFILE ... FORMAT AS PARQUET` now fails during BE writing. `ParquetFileFormatProperties` still defaults to Parquet 1.0 with INT96 disabled, and `VFileResultWriter` forwards those options to the generic writer. Its new Arrow `timestamp[ns]` reaches [Arrow 24's mandatory nanos-to-micros coercion](https://github.com/apache/arrow/blob/apache-arrow-24.0.0/cpp/src/parquet/column_writer.cc#L2398-L2408), which rejects the nonzero remainder as `would lose data`. The previous microsecond mapping exported successfully. Apply nanos-compatible version selection to generic OUTFILE, including nested fields, and test the default export/readback. The Iceberg-only version upgrade does not cover this caller; explicit `parquet.version=latest` currently works. This is a write-side failure separate from the existing legacy TVF/LOAD decoding thread.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](datalake) Support nanosecond timestamps in Iceberg and Paimon [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68934:
URL: https://github.com/apache/doris/pull/68934#issuecomment-6140414061
<details>
<summary>ClickBench: <b>Total hot run time: 24.49 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3f7020a33f405beaf37e5adbf07ccbb55a06774a, data reload: false
query1 0.01 0.00 0.01
query2 0.10 0.05 0.05
query3 0.28 0.12 0.11
query4 1.61 0.12 0.13
query5 0.27 0.25 0.25
query6 1.18 0.70 0.67
query7 0.04 0.01 0.00
query8 0.05 0.05 0.03
query9 0.41 0.35 0.34
query10 0.61 0.59 0.65
query11 0.22 0.14 0.14
query12 0.18 0.15 0.15
query13 0.51 0.49 0.49
query14 0.98 0.97 0.96
query15 0.63 0.61 0.61
query16 0.37 0.37 0.34
query17 1.11 1.13 1.14
query18 0.22 0.21 0.21
query19 2.08 1.96 1.93
query20 0.02 0.01 0.02
query21 15.49 0.27 0.15
query22 4.47 0.06 0.05
query23 16.29 0.32 0.13
query24 3.01 0.62 0.37
query25 0.12 0.06 0.06
query26 1.07 0.26 0.15
query27 0.05 0.04 0.03
query28 1.78 0.63 0.40
query29 12.61 4.20 3.28
query30 0.28 0.14 0.13
query31 2.77 0.57 0.34
query32 3.23 0.60 0.52
query33 3.17 3.27 3.31
query34 15.55 3.99 3.40
query35 3.34 3.34 3.37
query36 0.57 0.50 0.41
query37 0.09 0.06 0.07
query38 0.05 0.04 0.03
query39 0.04 0.03 0.02
query40 0.17 0.14 0.13
query41 0.09 0.03 0.03
query42 0.04 0.03 0.03
query43 0.05 0.04 0.03
Total cold run time: 95.21 s
Total hot run time: 24.49 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](datalake) Support nanosecond timestamps in Iceberg and Paimon [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #68934:
URL: https://github.com/apache/doris/pull/68934#issuecomment-6122692521
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](datalake) Support nanosecond timestamps in Iceberg and Paimon [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68934:
URL: https://github.com/apache/doris/pull/68934#issuecomment-6125135600
<details>
<summary>TPC-DS: <b>Total hot run time: 152217 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 643eb975c593ce69a4c3592b6a5769e053cb6b6f, data reload: false
query5 4305 608 461 461
query6 424 210 224 210
query7 4797 452 233 233
query8 323 178 165 165
query9 8774 4077 4069 4069
query10 452 311 261 261
query11 5981 3425 3197 3197
query12 140 86 88 86
query13 1240 441 333 333
query14 6562 4811 4548 4548
query14_1 4266 4273 4267 4267
query15 204 206 181 181
query16 942 447 432 432
query17 872 658 547 547
query18 2424 422 314 314
query19 201 164 140 140
query20 84 79 80 79
query21 215 138 118 118
query22 13060 13098 12826 12826
query23 13224 12549 12237 12237
query23_1 12312 12357 12139 12139
query24 7078 859 470 470
query24_1 472 478 472 472
query25 510 386 354 354
query26 1321 249 133 133
query27 2779 435 268 268
query28 4607 1982 1990 1982
query29 1566 592 448 448
query30 305 220 181 181
query31 873 757 647 647
query32 151 87 85 85
query33 499 299 241 241
query34 902 851 509 509
query35 745 757 662 662
query36 793 833 720 720
query37 126 105 90 90
query38 1808 1750 1698 1698
query39 720 741 704 704
query39_1 665 658 673 658
query40 222 125 98 98
query41 67 64 65 64
query42 88 86 81 81
query43 370 389 329 329
query44 1352 696 729 696
query45 179 176 162 162
query46 873 968 564 564
query47 2983 2659 2866 2659
query48 298 312 220 220
query49 573 391 321 321
query50 690 279 209 209
query51 10565 10431 10310 10310
query52 79 82 70 70
query53 191 212 151 151
query54 230 186 196 186
query55 72 74 65 65
query56 233 214 218 214
query57 1608 1578 1541 1541
query58 278 256 258 256
query59 2393 2419 2128 2128
query60 265 242 221 221
query61 147 144 136 136
query62 390 344 285 285
query63 198 159 170 159
query64 2732 928 825 825
query65 3456 3353 3353 3353
query66 1788 402 294 294
query67 20249 20069 20071 20069
query68 2939 1036 623 623
query69 393 297 258 258
query70 927 846 829 829
query71 307 231 218 218
query72 2723 2557 2307 2307
query73 525 553 297 297
query74 4528 4476 4294 4294
query75 2332 2290 1953 1953
query76 2300 1046 628 628
query77 359 403 302 302
query78 8931 8940 8453 8453
query79 902 865 520 520
query80 479 421 333 333
query81 518 327 275 275
query82 205 138 109 109
query83 198 195 178 178
query84 295 123 104 104
query85 762 408 355 355
query86 246 240 234 234
query87 2010 1947 1847 1847
query88 3643 2732 2719 2719
query89 340 294 259 259
query90 2097 187 189 187
query91 153 148 125 125
query92 99 79 86 79
query93 1024 1037 571 571
query94 448 317 278 278
query95 587 402 317 317
query96 681 555 245 245
query97 2434 2454 2386 2386
query98 154 148 140 140
query99 775 769 650 650
Total cold run time: 233322 ms
Total hot run time: 152217 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](datalake) Support nanosecond timestamps in Iceberg and Paimon [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #68934:
URL: https://github.com/apache/doris/pull/68934#issuecomment-6119754138
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](datalake) Support nanosecond timestamps in Iceberg and Paimon [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68934:
URL: https://github.com/apache/doris/pull/68934#issuecomment-6118174440
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://doris.apache.org/community/how-to-contribute/review-and-ci).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](datalake) Support nanosecond timestamps in Iceberg and Paimon [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #68934:
URL: https://github.com/apache/doris/pull/68934#issuecomment-6118175365
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](datalake) Support nanosecond timestamps in Iceberg and Paimon [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #68934:
URL: https://github.com/apache/doris/pull/68934#issuecomment-6137615765
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](datalake) Support nanosecond timestamps in Iceberg and Paimon [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68934:
URL: https://github.com/apache/doris/pull/68934#issuecomment-6137914313
<details>
<summary>TPC-H: <b>Total hot run time: 29434 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a48a522100127108c22a0319a9a82a7781e0b4c4, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17377 5680 5591 5591
q2 2030 318 255 255
q3 9845 1446 874 874
q4 4484 487 358 358
q5 7443 819 549 549
q6 202 191 156 156
q7 789 804 619 619
q8 10058 1363 1230 1230
q9 5836 4500 4520 4500
q10 6814 1293 1026 1026
q11 444 260 242 242
q12 507 415 305 305
q13 11946 3282 2399 2399
q14 275 260 253 253
q15 q16 732 730 680 680
q17 1195 757 594 594
q18 7083 6304 6199 6199
q19 709 981 626 626
q20 380 356 232 232
q21 3221 2486 2477 2477
q22 370 318 269 269
Total cold run time: 91740 ms
Total hot run time: 29434 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 6255 6210 6210 6210
q2 700 590 557 557
q3 5243 5480 5002 5002
q4 2075 2195 1616 1616
q5 5308 5187 5163 5163
q6 249 189 145 145
q7 2100 1891 1745 1745
q8 2982 2590 2614 2590
q9 7791 7736 7737 7736
q10 3731 3614 3290 3290
q11 616 444 385 385
q12 694 716 519 519
q13 2952 3235 2408 2408
q14 291 308 264 264
q15 q16 690 724 614 614
q17 7902 7032 6824 6824
q18 13192 12281 13102 12281
q19 974 856 910 856
q20 2219 2189 1953 1953
q21 6050 4952 4831 4831
q22 528 473 410 410
Total cold run time: 72542 ms
Total hot run time: 65399 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](datalake) Support nanosecond timestamps in Iceberg and Paimon [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68934:
URL: https://github.com/apache/doris/pull/68934#issuecomment-6138029096
<details>
<summary>TPC-DS: <b>Total hot run time: 152188 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a48a522100127108c22a0319a9a82a7781e0b4c4, data reload: false
query5 4286 616 469 469
query6 413 206 192 192
query7 4792 457 229 229
query8 314 210 164 164
query9 8773 4045 4010 4010
query10 441 316 255 255
query11 6054 3417 3198 3198
query12 154 87 83 83
query13 1233 447 323 323
query14 6542 4882 4550 4550
query14_1 4286 4261 4224 4224
query15 208 203 188 188
query16 940 463 409 409
query17 1047 632 529 529
query18 2407 409 306 306
query19 190 156 126 126
query20 78 76 75 75
query21 202 138 111 111
query22 12914 13100 12750 12750
query23 13253 12666 12235 12235
query23_1 12330 12166 12246 12166
query24 7072 848 459 459
query24_1 467 469 465 465
query25 507 396 338 338
query26 1242 253 139 139
query27 2771 445 278 278
query28 4607 1951 1952 1951
query29 1545 621 459 459
query30 299 223 186 186
query31 896 768 653 653
query32 143 106 92 92
query33 499 304 253 253
query34 964 873 499 499
query35 752 766 647 647
query36 841 815 743 743
query37 128 109 101 101
query38 1837 1773 1661 1661
query39 729 719 676 676
query39_1 659 674 672 672
query40 216 117 104 104
query41 74 69 70 69
query42 88 86 88 86
query43 373 383 330 330
query44 1355 712 714 712
query45 185 171 163 163
query46 851 975 573 573
query47 2971 3007 2881 2881
query48 307 325 236 236
query49 584 404 296 296
query50 703 299 221 221
query51 10435 10555 10100 10100
query52 84 82 72 72
query53 203 214 153 153
query54 243 219 217 217
query55 89 68 63 63
query56 242 222 215 215
query57 1623 1596 1539 1539
query58 287 266 261 261
query59 2376 2385 2136 2136
query60 286 269 231 231
query61 170 167 164 164
query62 400 348 293 293
query63 206 169 170 169
query64 2888 1088 757 757
query65 3469 3397 3404 3397
query66 1770 424 302 302
query67 20122 20428 20142 20142
query68 3288 1021 625 625
query69 384 285 259 259
query70 915 808 835 808
query71 296 248 209 209
query72 2854 2639 2270 2270
query73 534 550 304 304
query74 4665 4503 4338 4338
query75 2341 2313 1925 1925
query76 2376 1057 623 623
query77 420 394 302 302
query78 9125 9006 8516 8516
query79 981 936 533 533
query80 764 430 342 342
query81 583 312 280 280
query82 585 145 110 110
query83 289 197 180 180
query84 273 123 103 103
query85 943 417 364 364
query86 350 240 232 232
query87 2021 1985 1877 1877
query88 3690 2740 2708 2708
query89 337 300 272 272
query90 1703 190 188 188
query91 164 148 121 121
query92 93 90 78 78
query93 1025 1022 569 569
query94 499 305 285 285
query95 572 348 380 348
query96 678 545 238 238
query97 2454 2441 2323 2323
query98 156 146 154 146
query99 772 764 640 640
Total cold run time: 235221 ms
Total hot run time: 152188 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](datalake) Support nanosecond timestamps in Iceberg and Paimon [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #68934:
URL: https://github.com/apache/doris/pull/68934#issuecomment-6118348316
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](datalake) Support nanosecond timestamps in Iceberg and Paimon [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on code in PR #68934:
URL: https://github.com/apache/doris/pull/68934#discussion_r4255473949
##########
fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergPartitionUtils.java:
##########
@@ -801,6 +828,11 @@ static boolean isValidRelatedTable(Table table) {
return false;
}
PartitionField partitionField = fields.get(0);
+ if (spec.schema().findType(partitionField.sourceId()).typeId() == Type.TypeID.TIMESTAMP_NANO) {
Review Comment:
Confirmed at `3f7020a33f405beaf37e5adbf07ccbb55a06774a`. This is on the normal query path (`beginQuerySnapshot` → `latestSnapshotPin` → `isValidRelatedTable`), so the impact extends beyond MTMV and nanos tables. After the valid v1 remove-partition/drop-source lifecycle, the source type is absent and this dereference fails before the existing transform exclusion. Please treat an absent source type as ineligible for RANGE derivation and retain the UNPARTITIONED view, with a snapshot-planning test selecting a surviving column. Static source/dependency inspection only; I did not run tests.
##########
fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonTypeMapping.java:
##########
@@ -156,6 +156,10 @@ private static ConnectorType toDecimalType(DecimalType type) {
private static ConnectorType toTimestampType(DataType dataType) {
int scale = DEFAULT_TIMESTAMP_SCALE;
if (dataType instanceof TimestampType) {
+ // Keep all source fractional digits instead of truncating high-precision wall-clock values.
+ if (((TimestampType) dataType).getPrecision() > MAX_TIMESTAMP_SCALE) {
+ return ConnectorType.of("TIMESTAMP_NS");
Review Comment:
Confirmed at `3f7020a33f405beaf37e5adbf07ccbb55a06774a`: an existing Paimon `TIMESTAMP(7/8/9)` value such as `1000-01-01` now reaches `VectorColumn.appendTimestampNs` → `TypeNativeBytes.convertToTimestampNs`, where checked multiplication overflows. Previously the DATETIMEV2(6) projection retained access to that row. Please make the compatibility policy explicit rather than silently narrowing existing tables' readable range—for example, an explicit mapping option retaining the previous projection—and cover an actual out-of-range historical row. The current precision-7/8 truncation fix addresses fractional digits, but does not address this range change. Static inspection only; I did not run tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](datalake) Support nanosecond timestamps in Iceberg and Paimon [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68934:
URL: https://github.com/apache/doris/pull/68934#issuecomment-6145836781
<details>
<summary>TPC-H: <b>Total hot run time: 29344 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 19ef150201f4c4928c97b1efbe6003db6b83caf3, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17587 5668 5585 5585
q2 2104 313 291 291
q3 10220 1428 854 854
q4 4678 486 349 349
q5 7439 816 535 535
q6 188 195 161 161
q7 784 823 605 605
q8 9254 1371 1101 1101
q9 5877 4484 4503 4484
q10 6830 1301 1019 1019
q11 441 257 239 239
q12 633 420 296 296
q13 18044 3099 2424 2424
q14 277 277 257 257
q15 q16 751 736 684 684
q17 1214 748 573 573
q18 7180 6211 6167 6167
q19 1120 964 611 611
q20 380 343 240 240
q21 6015 2567 2712 2567
q22 410 390 302 302
Total cold run time: 101426 ms
Total hot run time: 29344 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 6931 6718 6778 6718
q2 712 617 567 567
q3 5518 5781 5238 5238
q4 2077 2214 1630 1630
q5 5865 5674 5812 5674
q6 256 189 152 152
q7 2250 2049 1809 1809
q8 3175 2815 2794 2794
q9 8151 8139 8052 8052
q10 3691 3620 3255 3255
q11 602 400 390 390
q12 680 706 512 512
q13 2779 3094 2411 2411
q14 297 282 261 261
q15 q16 672 676 624 624
q17 7798 6998 6931 6931
q18 13224 12290 13142 12290
q19 933 808 790 790
q20 2185 2186 1956 1956
q21 5873 4752 4853 4752
q22 506 467 408 408
Total cold run time: 74175 ms
Total hot run time: 67214 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](datalake) Support nanosecond timestamps in Iceberg and Paimon [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68934:
URL: https://github.com/apache/doris/pull/68934#issuecomment-6145866901
# FE Regression Coverage Report
Increment line coverage ` 2.65% (3/113)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/68934_19ef150201f4c4928c97b1efbe6003db6b83caf3_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/68934_19ef150201f4c4928c97b1efbe6003db6b83caf3_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](datalake) Support nanosecond timestamps in Iceberg and Paimon [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68934:
URL: https://github.com/apache/doris/pull/68934#issuecomment-6127378860
# FE UT Coverage Report
Increment line coverage ` 75.00% (3/4)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/68934_643eb975c593ce69a4c3592b6a5769e053cb6b6f/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/68934_643eb975c593ce69a4c3592b6a5769e053cb6b6f/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](datalake) Support nanosecond timestamps in Iceberg and Paimon [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68934:
URL: https://github.com/apache/doris/pull/68934#issuecomment-6183732223
<details>
<summary>ClickBench: <b>Total hot run time: 24.63 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 85d1412fccf4c085604a751c48e00dcce30e71b7, data reload: false
query1 0.01 0.01 0.00
query2 0.09 0.06 0.05
query3 0.27 0.12 0.11
query4 1.61 0.13 0.12
query5 0.24 0.25 0.26
query6 1.18 0.68 0.67
query7 0.04 0.01 0.00
query8 0.06 0.04 0.04
query9 0.42 0.33 0.35
query10 0.63 0.57 0.60
query11 0.22 0.14 0.16
query12 0.20 0.15 0.15
query13 0.49 0.49 0.49
query14 1.02 1.02 1.02
query15 0.63 0.62 0.60
query16 0.35 0.36 0.33
query17 1.15 1.16 1.12
query18 0.22 0.20 0.20
query19 2.10 2.00 2.01
query20 0.02 0.02 0.01
query21 15.50 0.25 0.15
query22 4.59 0.06 0.05
query23 16.27 0.33 0.12
query24 2.55 1.04 0.49
query25 0.12 0.06 0.04
query26 1.88 0.25 0.15
query27 0.05 0.03 0.04
query28 1.93 0.58 0.40
query29 12.54 4.12 3.29
query30 0.28 0.14 0.14
query31 2.78 0.58 0.34
query32 3.22 0.61 0.52
query33 3.35 3.23 3.18
query34 15.57 4.03 3.40
query35 3.34 3.34 3.34
query36 0.56 0.43 0.42
query37 0.10 0.07 0.07
query38 0.05 0.04 0.04
query39 0.04 0.03 0.03
query40 0.18 0.14 0.14
query41 0.09 0.03 0.03
query42 0.04 0.03 0.03
query43 0.04 0.03 0.03
Total cold run time: 96.02 s
Total hot run time: 24.63 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](datalake) Support nanosecond timestamps in Iceberg and Paimon [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #68934:
URL: https://github.com/apache/doris/pull/68934#issuecomment-6120288150
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](datalake) Support nanosecond timestamps in Iceberg and Paimon [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #68934:
URL: https://github.com/apache/doris/pull/68934#issuecomment-6120537950
Codex automated review failed and did not complete.
Error: Review recovery stopped: No final review submission was declared
Workflow run: https://github.com/apache/doris/actions/runs/38218043241
Please inspect the workflow logs and rerun the review after the underlying issue is resolved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](datalake) Support nanosecond timestamps in Iceberg and Paimon [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68934:
URL: https://github.com/apache/doris/pull/68934#issuecomment-6121177436
<details>
<summary>TPC-H: <b>Total hot run time: 28818 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5024d4e894591b18ef797542542aa1b040d73696, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17582 5700 5542 5542
q2 2070 313 248 248
q3 10269 1414 847 847
q4 4683 483 357 357
q5 7435 799 529 529
q6 189 195 159 159
q7 766 802 611 611
q8 9261 1326 1122 1122
q9 5688 4536 4441 4441
q10 6811 1302 1001 1001
q11 442 262 237 237
q12 628 404 286 286
q13 18070 3101 2412 2412
q14 270 279 257 257
q15 q16 745 741 668 668
q17 1185 742 565 565
q18 7173 6231 6166 6166
q19 1135 951 603 603
q20 371 337 229 229
q21 4793 2262 2429 2262
q22 361 328 276 276
Total cold run time: 99927 ms
Total hot run time: 28818 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 6149 6109 6104 6104
q2 706 574 562 562
q3 5227 5415 4945 4945
q4 2025 2131 1555 1555
q5 5268 5201 6017 5201
q6 256 195 155 155
q7 2291 2042 1872 1872
q8 3157 2739 2744 2739
q9 8227 8294 8474 8294
q10 3793 3669 3315 3315
q11 644 433 427 427
q12 711 735 575 575
q13 2886 3218 2506 2506
q14 319 309 312 309
q15 q16 709 773 646 646
q17 8437 7440 7408 7408
q18 13183 12249 13102 12249
q19 938 790 796 790
q20 2194 2142 1947 1947
q21 5763 4690 4758 4690
q22 512 450 395 395
Total cold run time: 73395 ms
Total hot run time: 66684 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](datalake) Support nanosecond timestamps in Iceberg and Paimon [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #68934:
URL: https://github.com/apache/doris/pull/68934#issuecomment-6122693055
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](datalake) Support nanosecond timestamps in Iceberg and Paimon [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68934:
URL: https://github.com/apache/doris/pull/68934#issuecomment-6144767938
<details>
<summary>ClickBench: <b>Total hot run time: 24.67 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 19ef150201f4c4928c97b1efbe6003db6b83caf3, data reload: false
query1 0.01 0.01 0.01
query2 0.11 0.05 0.05
query3 0.27 0.12 0.12
query4 1.61 0.12 0.12
query5 0.27 0.25 0.24
query6 1.17 0.69 0.68
query7 0.04 0.01 0.01
query8 0.06 0.04 0.03
query9 0.41 0.35 0.35
query10 0.59 0.60 0.60
query11 0.21 0.14 0.14
query12 0.19 0.14 0.14
query13 0.48 0.50 0.48
query14 0.98 0.97 0.96
query15 0.63 0.61 0.62
query16 0.34 0.32 0.34
query17 1.15 1.19 1.15
query18 0.22 0.21 0.21
query19 2.15 2.05 2.00
query20 0.02 0.01 0.02
query21 15.51 0.28 0.14
query22 4.41 0.06 0.05
query23 16.29 0.32 0.13
query24 3.13 0.69 0.40
query25 0.12 0.06 0.04
query26 1.34 0.25 0.14
query27 0.05 0.03 0.04
query28 2.15 0.57 0.39
query29 12.50 4.19 3.29
query30 0.29 0.14 0.13
query31 2.77 0.58 0.35
query32 3.23 0.62 0.52
query33 3.14 3.33 3.28
query34 15.48 4.02 3.40
query35 3.35 3.35 3.38
query36 0.56 0.45 0.45
query37 0.10 0.06 0.06
query38 0.05 0.03 0.03
query39 0.04 0.03 0.04
query40 0.16 0.15 0.14
query41 0.08 0.03 0.03
query42 0.04 0.03 0.03
query43 0.03 0.03 0.04
Total cold run time: 95.73 s
Total hot run time: 24.67 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](datalake) Support nanosecond timestamps in Iceberg and Paimon [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68934:
URL: https://github.com/apache/doris/pull/68934#issuecomment-6146102390
<details>
<summary>ClickBench: <b>Total hot run time: 24.45 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 19ef150201f4c4928c97b1efbe6003db6b83caf3, data reload: false
query1 0.01 0.01 0.00
query2 0.10 0.05 0.05
query3 0.27 0.12 0.11
query4 1.62 0.12 0.12
query5 0.28 0.22 0.22
query6 1.17 0.70 0.68
query7 0.05 0.01 0.01
query8 0.07 0.04 0.04
query9 0.41 0.36 0.36
query10 0.63 0.58 0.60
query11 0.21 0.15 0.14
query12 0.19 0.14 0.15
query13 0.50 0.50 0.49
query14 0.99 0.98 0.96
query15 0.63 0.62 0.61
query16 0.34 0.36 0.34
query17 1.15 1.15 1.13
query18 0.23 0.20 0.20
query19 2.11 1.94 1.95
query20 0.02 0.01 0.02
query21 15.50 0.29 0.14
query22 4.39 0.06 0.06
query23 16.27 0.32 0.12
query24 3.18 0.63 0.37
query25 0.10 0.05 0.05
query26 1.00 0.26 0.15
query27 0.05 0.04 0.04
query28 1.90 0.66 0.41
query29 12.50 4.14 3.30
query30 0.28 0.14 0.13
query31 2.77 0.59 0.33
query32 3.23 0.62 0.53
query33 3.15 3.13 3.20
query34 15.43 4.02 3.40
query35 3.38 3.36 3.38
query36 0.56 0.48 0.44
query37 0.09 0.06 0.07
query38 0.05 0.04 0.03
query39 0.03 0.04 0.03
query40 0.18 0.14 0.15
query41 0.08 0.03 0.03
query42 0.03 0.03 0.04
query43 0.04 0.04 0.04
Total cold run time: 95.17 s
Total hot run time: 24.45 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](datalake) Support nanosecond timestamps in Iceberg and Paimon [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on code in PR #68934:
URL: https://github.com/apache/doris/pull/68934#discussion_r4256215913
##########
fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorTransaction.java:
##########
@@ -929,14 +930,19 @@ private Expression buildPartitionFilter(Map<String, String> staticPartitions, Pa
* both paths that build this predicate: DELETE/UPDATE/MERGE conflict detection and
* {@code INSERT OVERWRITE ... PARTITION(d='nan')}.
*/
- private static Expression identityPartitionPredicate(String sourceColName, Object value) {
+ private static Expression identityPartitionPredicate(Types.NestedField sourceField, Object value) {
+ String sourceColName = sourceField.name();
if (value == null) {
return Expressions.isNull(sourceColName);
}
if ((value instanceof Double || value instanceof Float)
&& Double.isNaN(((Number) value).doubleValue())) {
return Expressions.isNaN(sourceColName);
}
+ if (sourceField.type().typeId() == TypeID.TIMESTAMP_NANO) {
+ // Partition values are already nanos; a raw Long would be rebound as microseconds.
+ return Expressions.equal(sourceColName, Expressions.nanos((Long) value));
Review Comment:
Fixed in 19ef150201f4. Generated overwrite and conflict predicates now share the same table-history safety check as scan predicates. Static overwrite rejects a non-null nanos partition predicate when retained DATE bounds make deletion unsafe; it does not broaden deletion. Conflict-only filters omit unsafe conjuncts conservatively. The new transaction test retains an unpartitioned DATE file, promotes its field, adds an identity partition, and verifies that the attempted overwrite fails without changing the snapshot. The 425-test FE selection passes.
##########
fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergPredicateConverter.java:
##########
@@ -475,7 +507,10 @@ private Types.NestedField getPushdownField(String colName) {
|| ICEBERG_LAST_UPDATED_SEQUENCE_NUMBER_COL.equalsIgnoreCase(colName)) {
return null;
}
- return schema.caseInsensitiveFindField(colName);
+ Types.NestedField field = schema.caseInsensitiveFindField(colName);
+ // Iceberg 1.11 widens retained four-byte DATE bounds to unscaled longs. Pruning those
+ // epoch-day bounds with nanosecond predicates can discard matching files in every planner.
+ return field != null && unsafeNanosFields.contains(field.fieldId()) ? null : field;
Review Comment:
Fixed in 19ef150201f4. NOT now requires exact conversion of its entire operand, so an unsafe child cannot turn NOT(ts = ... AND id > 0) into NOT(id > 0). Tests exercise the unsimplified ConnectorNot directly and verify file retention through both SDK and manifest-cache planning; safe negation remains pushable. The 425-test FE selection passes.
##########
fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergPartitionUtils.java:
##########
@@ -801,6 +828,11 @@ static boolean isValidRelatedTable(Table table) {
return false;
}
PartitionField partitionField = fields.get(0);
+ if (spec.schema().findType(partitionField.sourceId()).typeId() == Type.TypeID.TIMESTAMP_NANO) {
Review Comment:
Fixed in 19ef150201f4. A missing historical partition source type now makes the table ineligible for Doris RANGE derivation rather than dereferencing null. The test creates an actual Iceberg v1 table, removes day(ts), and drops ts, then verifies the retained historical/void specs are handled. This is Iceberg metadata handling, distinct from the excluded Doris format-v1 reader. The 425-test FE selection passes.
##########
be/src/core/data_type_serde/data_type_datetimev2_serde.cpp:
##########
@@ -1049,7 +1050,11 @@ Status DataTypeDateTimeV2SerDe::read_column_from_orc(IColumn& column,
if (orc_serde_utils::orc_decode_row_count(view.rows, view.selected_rows) == 0) {
return Status::OK();
}
- return decode_timestamp_orc_values(column, view, *view.timezone);
+ const auto* timezone = view.file_type->getKind() == ::orc::TIMESTAMP &&
+ view.plain_timestamp_timezone != nullptr
+ ? view.plain_timestamp_timezone
Review Comment:
Fixed in 19ef150201f4 without modifying the SDK. The Doris decode context uses the pinned SDK writer-zone rules for Java ORC carriers and preserves cctz historical civil-time semantics for C++ carriers. Writer offsets are cached in the per-reader context. A checked-in fixture generated by Java ORC 1.9.0 covers year 1800 and epoch-boundary fractions, with the NANOS sibling both projected and hidden, across UTC, Shanghai, and New York sessions. Existing C++ historical/lazy/row-ID cases also pass in the 949-test BE selection.
##########
be/src/format_v2/orc/orc_reader.cpp:
##########
@@ -1646,24 +1715,91 @@ Status OrcReader::_select_stripe_ranges_by_statistics() {
return Status::OK();
}
-void OrcReader::_apply_current_stripe_range() {
- if (!_state->stripe_pruning_applied || _state->selected_stripe_ranges.empty()) {
- return;
+Status OrcReader::_prepare_timestamp_stripe_ranges() {
+ if (!_state->match_writer_timezone) {
+ return Status::OK();
+ }
+ std::vector<uint64_t> stripes;
+ if (_state->stripe_ranges_applied) {
+ for (const auto& range : _state->selected_stripe_ranges) {
+ for (auto stripe = range.first_stripe; stripe < range.last_stripe; ++stripe) {
+ stripes.push_back(stripe);
+ }
+ }
+ } else {
+ RETURN_IF_ERROR(_collect_split_stripes(&stripes));
+ }
+ _state->selected_stripe_ranges.clear();
+ _state->stripe_ranges_applied = true;
+ _reader_statistics.read_row_groups = cast_set<int32_t>(stripes.size());
+ if (stripes.empty()) {
+ return Status::OK();
+ }
+ try {
+ uint64_t first_row = 0;
+ size_t selected = 0;
+ for (uint64_t index = 0; index <= stripes.back(); ++index) {
+ const auto stripe = _state->reader->getStripe(index);
+ const auto end_row = first_row + stripe->getNumberOfRows();
+ if (index == stripes[selected]) {
+ // A RowReader has one timezone, whereas writer timezones belong to stripes.
+ // Read footer timezone metadata only when entering the selected stripe.
+ _state->selected_stripe_ranges.push_back({
+ .first_stripe = index,
+ .last_stripe = index + 1,
+ .offset = stripe->getOffset(),
+ .length = stripe->getLength(),
+ .first_row = first_row,
+ .end_row = end_row,
+ });
+ ++selected;
+ }
+ first_row = end_row;
+ }
+ } catch (const std::exception& e) {
+ return Status::InternalError("Failed to prepare ORC timestamp stripe ranges: {}", e.what());
+ }
+ return Status::OK();
+}
+
+Status OrcReader::_apply_current_stripe_range() {
+ if (!_state->stripe_ranges_applied || _state->selected_stripe_ranges.empty()) {
+ return Status::OK();
}
DORIS_CHECK(_state->current_stripe_range < _state->selected_stripe_ranges.size());
const auto& stripe_range = _state->selected_stripe_ranges[_state->current_stripe_range];
_state->row_reader_options.range(stripe_range.offset, stripe_range.length);
+ if (_state->match_writer_timezone) {
+ try {
+ const auto stripe = _state->reader->getStripe(stripe_range.first_stripe);
+ const auto& name = stripe->getWriterTimezone();
+ const auto writer_timezone = name.empty() ? "GMT" : name;
+ if (!TimezoneUtils::find_cctz_time_zone(writer_timezone, _state->writer_timezone_obj)) {
+ return Status::InvalidArgument("Unknown ORC stripe writer timezone: {}",
+ writer_timezone);
+ }
+ // Matching zones preserves the SDK's UTC carrier until its negative-fraction
+ // correction is complete. cctz then applies historical offsets to that original second.
+ _state->row_reader_options.setTimezoneName(writer_timezone);
Review Comment:
Fixed in 19ef150201f4. Metadata MIN/MAX now declines the unsafe C++ named-zone mixed-NANOS plain-TIMESTAMP case so aggregation reads the rows; fixed-zone writers keep the metadata optimization. The historical C++ fixture now uses a nonzero .123456789 fraction, verifies row values, and verifies that metadata aggregation requests fall back instead of returning the inconsistent bound. The 949-test BE selection passes.
##########
be/src/format_v2/orc/orc_reader.cpp:
##########
@@ -1646,24 +1715,91 @@ Status OrcReader::_select_stripe_ranges_by_statistics() {
return Status::OK();
}
-void OrcReader::_apply_current_stripe_range() {
- if (!_state->stripe_pruning_applied || _state->selected_stripe_ranges.empty()) {
- return;
+Status OrcReader::_prepare_timestamp_stripe_ranges() {
+ if (!_state->match_writer_timezone) {
+ return Status::OK();
+ }
+ std::vector<uint64_t> stripes;
+ if (_state->stripe_ranges_applied) {
+ for (const auto& range : _state->selected_stripe_ranges) {
+ for (auto stripe = range.first_stripe; stripe < range.last_stripe; ++stripe) {
+ stripes.push_back(stripe);
+ }
+ }
+ } else {
+ RETURN_IF_ERROR(_collect_split_stripes(&stripes));
+ }
+ _state->selected_stripe_ranges.clear();
+ _state->stripe_ranges_applied = true;
+ _reader_statistics.read_row_groups = cast_set<int32_t>(stripes.size());
+ if (stripes.empty()) {
+ return Status::OK();
+ }
+ try {
+ uint64_t first_row = 0;
+ size_t selected = 0;
+ for (uint64_t index = 0; index <= stripes.back(); ++index) {
+ const auto stripe = _state->reader->getStripe(index);
+ const auto end_row = first_row + stripe->getNumberOfRows();
+ if (index == stripes[selected]) {
+ // A RowReader has one timezone, whereas writer timezones belong to stripes.
+ // Read footer timezone metadata only when entering the selected stripe.
+ _state->selected_stripe_ranges.push_back({
+ .first_stripe = index,
+ .last_stripe = index + 1,
Review Comment:
Fixed in 19ef150201f4 without SDK changes. Adjacent selected stripes with the same effective writer timezone share one RowReader; gaps and timezone changes remain separate ranges. Added OrcRowReadersCreated profiling and a 64-stripe NANOS test that checks all values and exactly one RowReader. Historical timestamp tests also check reader reuse across lazy reads, row-ID fetch, and pruning. The 949-test BE selection passes.
##########
be/src/format_v2/parquet/native_schema_desc.cpp:
##########
@@ -882,6 +882,11 @@ std::pair<DataTypePtr, bool> NativeFieldDescriptor::convert_to_doris_type(
// sub-second values are silently truncated by the target SerDe.
ans.first = DataTypeFactory::instance().create_data_type(TYPE_TIMEV2, nullable, 0, scale);
} else if (logicalType.__isset.TIMESTAMP) {
+ if (logicalType.TIMESTAMP.unit.__isset.NANOS && !logicalType.TIMESTAMP.isAdjustedToUTC) {
+ // Materialize nanoseconds before any table cast; DATETIMEV2 would discard the remainder.
+ ans.first = DataTypeFactory::instance().create_data_type(TYPE_TIMESTAMP_NS, nullable);
Review Comment:
Fixed in 19ef150201f4. ColumnTimeStampNs is registered in the existing nullable in-place expansion dispatcher. A filtered alternating-null NANOS test covers both plain and dictionary pages, checks negative/submicrosecond values and the null map, and asserts HybridSelectionNullFallbackBatches remains zero. The test failed before the dispatch fix and passes in the 949-test BE selection.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](datalake) Support nanosecond timestamps in Iceberg and Paimon [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on code in PR #68934:
URL: https://github.com/apache/doris/pull/68934#discussion_r4254435732
##########
fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergTypeMapping.java:
##########
@@ -150,12 +150,16 @@ private static ConnectorType fromPrimitive(Type.PrimitiveType primitive,
return ConnectorType.of("TIMESTAMPTZ", ICEBERG_DATETIME_SCALE_MS, 0);
}
return ConnectorType.of("DATETIMEV2", ICEBERG_DATETIME_SCALE_MS, 0);
+ case TIMESTAMP_NANO:
+ // TIMESTAMP_NS is a wall-clock type; zoned nanoseconds need a distinct carrier.
+ return ConnectorType.of(((Types.TimestampNanoType) primitive).shouldAdjustToUTC()
+ ? "UNSUPPORTED" : "TIMESTAMP_NS");
Review Comment:
Reworked the promotion policy into a dedicated `format::EvolutionCast` expression. Ordinary `Cast` no longer has an `is_schema_evolution` flag or an external-format validation branch. `EvolutionCast` reuses the existing checked conversion and owns the DATEV2-to-TIMESTAMP_NS strict policy; the execution-layer CAST functions and ordinary SQL CAST semantics are unchanged.
Projection, localized filters, table materialization, and equality-delete/default conversions now create `EvolutionCast`. Both expression-tree cloning and runtime-context cloning preserve the check.
Local validation: 384 BE tests passed, including seven focused expression tests and retained Parquet DATE files with scalar/nested projections and localized filtering. The tests cover both strict-cast settings, range boundaries, NULLs, constant folding, empty input, numeric promotion, and cloning.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](datalake) Support nanosecond timestamps in Iceberg and Paimon [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68934:
URL: https://github.com/apache/doris/pull/68934#issuecomment-6185597624
# BE Regression && UT Coverage Report
Increment line coverage `92.13% (82/89)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/68934_85d1412fccf4c085604a751c48e00dcce30e71b7_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/68934_85d1412fccf4c085604a751c48e00dcce30e71b7_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 76.98% (35605/46250) |
| Line Coverage | 62.32% (403553/647502) |
| Region Coverage | 58.76% (340984/580268) |
| Branch Coverage | 59.60% (156969/263360) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](datalake) Support nanosecond timestamps in Iceberg and Paimon [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 merged PR #68934:
URL: https://github.com/apache/doris/pull/68934
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org