You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "eldenmoon (via GitHub)" <gi...@apache.org> on 2026/05/17 09:38:37 UTC
[PR] [fix](be) Preserve Variant decimal precision [doris]
eldenmoon opened a new pull request, #63322:
URL: https://github.com/apache/doris/pull/63322
### What problem does this PR solve?
Issue Number: CIR-20240
Related PR: None
Problem Summary: Variant JSON parsing materialized high precision decimal-like numbers as double before typed decimal paths were cast, which lost precision. Preserve raw numeric tokens for decimal-sensitive Variant path extraction and cast typed decimal paths from exact text while keeping Variant nullable array structure.
### Release note
Fix Variant typed decimal precision loss when parsing high precision JSON numbers.
### Check List (For Author)
- Test:
- Build: BUILD_TYPE=ASAN USE_MEM_TRACKER=ON ./build.sh --be --fe
- Unit Test: ./run-be-ut.sh --run --filter='JsonParserTest.*:ColumnVariantTest.parse_high_precision_decimal_for_typed_path'
- Regression test: env -u http_proxy -u https_proxy -u HTTP_PROXY -u HTTPS_PROXY -u all_proxy -u ALL_PROXY NO_PROXY=127.0.0.1,localhost,10.26.20.3 no_proxy=127.0.0.1,localhost,10.26.20.3 ./run-regression-test.sh --conf tmp/regression-conf.auto.groovy --run -d variant_p0/predefine -s test_variant_high_precision_decimal
- Format Check: build-support/clang-format.sh; build-support/check-format.sh
- Static Check: build-support/run-clang-tidy.sh --build-dir be/ut_build_ASAN (attempted; failed due existing/environment diagnostics including missing stddef.h and unrelated existing readability-function-size warnings)
- Behavior changed: Yes (Variant typed decimal paths cast preserved raw JSON number text instead of rounded double for precision-sensitive numeric tokens)
- Does this need documentation: No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant predefined decimal precision [doris]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4475102521
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant predefined decimal precision [doris]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4473927372
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant predefined decimal precision [doris]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4474149636
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant decimal precision [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4470942286
<details>
<summary>TPC-DS: <b>Total hot run time: 169474 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3dd972bebfba3e772374972a1829da943aea8361, data reload: false
query5 4320 659 536 536
query6 336 222 197 197
query7 4241 576 311 311
query8 328 237 233 233
query9 8804 4071 4011 4011
query10 443 337 290 290
query11 5770 2421 2270 2270
query12 185 133 129 129
query13 1303 581 419 419
query14 5931 5417 4990 4990
query14_1 4338 4323 4334 4323
query15 215 202 180 180
query16 1026 454 413 413
query17 1135 714 588 588
query18 2713 480 349 349
query19 222 205 155 155
query20 142 131 131 131
query21 213 143 115 115
query22 13752 13636 13340 13340
query23 17114 16388 16009 16009
query23_1 16245 16194 16140 16140
query24 7494 1776 1321 1321
query24_1 1335 1331 1302 1302
query25 576 513 442 442
query26 1101 327 176 176
query27 2691 548 349 349
query28 4465 1929 1972 1929
query29 1000 647 528 528
query30 307 252 200 200
query31 1129 1081 960 960
query32 100 80 74 74
query33 565 366 307 307
query34 1166 1162 646 646
query35 768 792 677 677
query36 1368 1375 1215 1215
query37 154 109 96 96
query38 3233 3126 3074 3074
query39 923 926 919 919
query39_1 887 879 877 877
query40 232 156 132 132
query41 72 70 70 70
query42 111 112 111 111
query43 326 331 287 287
query44
query45 216 207 205 205
query46 1072 1187 754 754
query47 2358 2370 2198 2198
query48 417 405 304 304
query49 650 513 417 417
query50 974 347 257 257
query51 4294 4312 4240 4240
query52 110 110 98 98
query53 258 281 208 208
query54 333 286 280 280
query55 95 93 93 93
query56 315 319 318 318
query57 1416 1420 1315 1315
query58 321 282 268 268
query59 1623 1656 1508 1508
query60 333 337 329 329
query61 190 221 158 158
query62 663 625 570 570
query63 247 207 210 207
query64 2105 815 649 649
query65
query66 1638 473 366 366
query67 29998 29979 29816 29816
query68
query69 452 341 320 320
query70 1067 1017 1018 1017
query71 309 280 264 264
query72 3016 2732 2459 2459
query73 873 760 398 398
query74 5070 4896 4726 4726
query75 2664 2593 2246 2246
query76 2288 1133 770 770
query77 390 415 338 338
query78 12247 12205 11669 11669
query79 1460 1027 745 745
query80 1297 547 452 452
query81 507 281 238 238
query82 1238 167 125 125
query83 318 276 249 249
query84 274 144 110 110
query85 939 528 450 450
query86 448 319 315 315
query87 3430 3441 3209 3209
query88 3526 2677 2673 2673
query89 449 383 341 341
query90 1884 183 181 181
query91 179 177 165 165
query92 80 78 71 71
query93 1472 1422 874 874
query94 726 346 307 307
query95 678 477 353 353
query96 1025 794 358 358
query97 2724 2706 2561 2561
query98 238 232 226 226
query99 1147 1108 1002 1002
Total cold run time: 253890 ms
Total hot run time: 169474 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant predefined decimal precision [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #63322:
URL: https://github.com/apache/doris/pull/63322#discussion_r3256689390
##########
be/src/core/data_type_serde/data_type_variant_serde.cpp:
##########
@@ -108,11 +104,11 @@ Status DataTypeVariantSerDe::serialize_one_cell_to_json(const IColumn& column, i
Status DataTypeVariantSerDe::deserialize_one_cell_from_json(IColumn& column, Slice& slice,
const FormatOptions& options) const {
- ParseConfig parse_config;
- parse_config.check_duplicate_json_path = config::variant_enable_duplicate_json_path_check;
- StringRef json_ref(slice.data, slice.size);
- RETURN_IF_CATCH_EXCEPTION(
- variant_util::parse_json_to_variant(column, json_ref, nullptr, parse_config));
+ auto& variant = assert_cast<ColumnVariant&>(column);
+ Field field = Field::create_field<TYPE_STRING>(String(slice.data, slice.size));
+ VariantMap object;
Review Comment:
This serde is also used by the generic cast path, not only by stream load. `cast_from_string_to_generic()` calls `serde->deserialize_one_cell_from_json()` for `CAST(string AS VARIANT)`, and existing suites rely on `cast('{"a":1}' as variant)['a']` exposing subpaths. With this change the same input is always inserted as a root `TYPE_STRING`, so query-time casts no longer parse the JSON object and subpath access like `c1['a']` returns missing/null instead of `1`. This is distinct from the direct Variant stream-load concern because it affects expression evaluation without the later storage-layer `parse_and_materialize_variant_columns()` pass. Please keep the old parse behavior for casts/generic serde callers and restrict raw-root preservation to the load path that can materialize later, and add a regression for `cast('{"a":1}' as variant)['a']`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant decimal precision [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #63322:
URL: https://github.com/apache/doris/pull/63322#discussion_r3254617911
##########
be/src/format/json/new_json_reader.cpp:
##########
@@ -1139,6 +1139,16 @@ Status NewJsonReader::_simdjson_write_data_to_column(simdjson::ondemand::value&
}
auto primitive_type = type_desc->get_primitive_type();
+ if (_is_load && primitive_type == TYPE_VARIANT) {
+ std::string_view json_str = simdjson::to_json_string(value);
+ Slice slice {json_str.data(), json_str.size()};
+ auto variant_options = _serde_options;
+ variant_options.variant_read_raw_json = true;
+ RETURN_IF_ERROR(data_serde->deserialize_one_cell_from_json(*data_column_ptr, slice,
+ variant_options));
+ return Status::OK();
Review Comment:
This early return skips the common nullable-column tail at lines 1326-1329, so a non-null value loaded into a nullable `VARIANT` column inserts one row into the nested `ColumnVariant` but never appends `0` to the `ColumnNullable` null map. A normal stream load into a schema like the new regression table (`v variant ... NULL`) will leave the nullable column with mismatched nested/null-map sizes, which can trip consistency checks or corrupt later reads. Please keep the raw-Variant branch on the existing path that reaches the final null-map append, or append the null-map entry before returning, and add a nullable Variant stream-load case that validates the column remains consistent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant decimal precision [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4471565342
<details>
<summary>TPC-H: <b>Total hot run time: 31453 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b2ec8138ab58c2542048d496e500a78733c50214, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17610 3863 3847 3847
q2 q3 10801 1346 812 812
q4 4689 479 342 342
q5 7605 2333 2078 2078
q6 238 180 139 139
q7 956 785 630 630
q8 9361 1803 1662 1662
q9 5209 4925 4915 4915
q10 6385 2072 1814 1814
q11 435 266 245 245
q12 624 424 299 299
q13 18134 3414 2779 2779
q14 266 253 238 238
q15 q16 820 772 712 712
q17 936 866 950 866
q18 6898 5765 5551 5551
q19 1313 1383 1094 1094
q20 528 406 367 367
q21 6266 2915 2740 2740
q22 463 376 323 323
Total cold run time: 99537 ms
Total hot run time: 31453 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4777 4539 4596 4539
q2 q3 4838 5259 4639 4639
q4 2122 2225 1407 1407
q5 4758 4830 4716 4716
q6 234 176 129 129
q7 1869 1805 1527 1527
q8 2374 2042 2030 2030
q9 7614 7484 7206 7206
q10 4489 4410 3998 3998
q11 523 383 347 347
q12 700 714 509 509
q13 3051 3376 2797 2797
q14 284 273 249 249
q15 q16 678 699 605 605
q17 1260 1256 1240 1240
q18 7507 6778 6871 6778
q19 1118 1090 1073 1073
q20 2200 2219 1944 1944
q21 5318 4682 4501 4501
q22 522 469 411 411
Total cold run time: 56236 ms
Total hot run time: 50645 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant decimal precision [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4470736287
<details>
<summary>TPC-H: <b>Total hot run time: 31185 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 85fcf569e2cba61ca79f53700ba241f8851c2bd3, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17718 3918 3857 3857
q2 q3 10842 1415 838 838
q4 4713 478 342 342
q5 8135 2249 2134 2134
q6 339 179 136 136
q7 981 794 649 649
q8 9406 1710 1619 1619
q9 7007 4996 4893 4893
q10 6448 2147 1830 1830
q11 439 277 242 242
q12 693 426 295 295
q13 18184 3414 2777 2777
q14 276 257 230 230
q15 q16 824 763 709 709
q17 963 945 959 945
q18 7005 5736 5602 5602
q19 1167 1237 1123 1123
q20 509 416 264 264
q21 5698 2620 2405 2405
q22 428 360 295 295
Total cold run time: 101775 ms
Total hot run time: 31185 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4200 4126 4123 4123
q2 q3 4485 4902 4284 4284
q4 2099 2212 1395 1395
q5 4391 4261 4644 4261
q6 264 196 145 145
q7 2018 1850 1645 1645
q8 2440 2112 2168 2112
q9 7799 7811 7688 7688
q10 4508 4461 4238 4238
q11 576 410 383 383
q12 788 732 510 510
q13 3267 3675 2930 2930
q14 301 315 291 291
q15 q16 723 745 655 655
q17 1321 1349 1311 1311
q18 7849 7407 6706 6706
q19 1158 1079 1076 1076
q20 2228 2206 1924 1924
q21 5315 4650 4477 4477
q22 534 474 407 407
Total cold run time: 56264 ms
Total hot run time: 50561 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant decimal precision [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4471220081
<details>
<summary>TPC-DS: <b>Total hot run time: 169031 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 149c7f4f34ee4e10b9562d7dfea1bc11ad5227ea, data reload: false
query5 4350 668 515 515
query6 331 214 200 200
query7 4238 546 317 317
query8 319 223 213 213
query9 8807 4036 4055 4036
query10 454 336 301 301
query11 5773 2359 2201 2201
query12 180 129 129 129
query13 1288 629 456 456
query14 5963 5357 5025 5025
query14_1 4335 4363 4347 4347
query15 207 199 183 183
query16 1013 436 427 427
query17 1157 716 583 583
query18 2487 495 351 351
query19 211 203 162 162
query20 145 139 129 129
query21 217 141 116 116
query22 13628 13493 13452 13452
query23 17193 16336 16103 16103
query23_1 16165 16204 16152 16152
query24 7431 1797 1309 1309
query24_1 1320 1318 1310 1310
query25 578 505 440 440
query26 1308 329 177 177
query27 2685 546 348 348
query28 4454 1958 1959 1958
query29 1044 661 536 536
query30 323 240 204 204
query31 1145 1055 935 935
query32 85 73 75 73
query33 538 352 287 287
query34 1172 1149 652 652
query35 756 781 681 681
query36 1326 1363 1157 1157
query37 158 100 88 88
query38 3186 3137 3038 3038
query39 913 935 893 893
query39_1 861 881 881 881
query40 231 154 125 125
query41 67 63 63 63
query42 112 113 109 109
query43 332 330 296 296
query44
query45 208 204 195 195
query46 1073 1175 713 713
query47 2337 2385 2193 2193
query48 400 413 292 292
query49 627 501 386 386
query50 964 342 256 256
query51 4352 4406 4255 4255
query52 105 108 97 97
query53 264 298 211 211
query54 326 291 252 252
query55 93 96 90 90
query56 326 321 306 306
query57 1436 1433 1330 1330
query58 303 288 260 260
query59 1628 1662 1418 1418
query60 332 327 320 320
query61 155 156 155 155
query62 650 641 569 569
query63 247 205 209 205
query64 2400 833 626 626
query65
query66 1708 485 359 359
query67 29340 29349 29865 29349
query68
query69 457 345 305 305
query70 1022 984 956 956
query71 326 279 273 273
query72 2922 2705 2426 2426
query73 833 761 417 417
query74 5088 4878 4743 4743
query75 2673 2577 2248 2248
query76 2270 1139 798 798
query77 400 415 338 338
query78 12134 11924 11575 11575
query79 1235 1046 734 734
query80 602 541 461 461
query81 446 283 240 240
query82 242 157 120 120
query83 270 277 248 248
query84 253 131 111 111
query85 847 519 436 436
query86 359 365 308 308
query87 3378 3348 3263 3263
query88 3540 2654 2662 2654
query89 421 387 338 338
query90 2181 185 184 184
query91 184 168 141 141
query92 79 77 76 76
query93 1439 1419 891 891
query94 546 349 302 302
query95 659 376 412 376
query96 997 817 344 344
query97 2697 2695 2585 2585
query98 232 225 227 225
query99 1094 1113 968 968
Total cold run time: 250564 ms
Total hot run time: 169031 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant decimal precision [doris]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4471220489
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant predefined decimal precision [doris]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4474705442
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant decimal precision [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4470934918
<details>
<summary>TPC-DS: <b>Total hot run time: 169122 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9a398bd168d5dc2f4047719bccd4b533f710ea0b, data reload: false
query5 4327 663 528 528
query6 325 218 204 204
query7 4234 542 296 296
query8 322 258 224 224
query9 8843 4016 4009 4009
query10 446 336 293 293
query11 5827 2389 2190 2190
query12 202 128 129 128
query13 1249 587 425 425
query14 5912 5336 5033 5033
query14_1 4299 4318 4299 4299
query15 209 201 186 186
query16 998 441 428 428
query17 931 697 580 580
query18 2443 482 350 350
query19 224 211 183 183
query20 133 137 132 132
query21 218 141 115 115
query22 13589 13495 13368 13368
query23 17306 16304 16104 16104
query23_1 16244 16166 16161 16161
query24 7520 1769 1307 1307
query24_1 1279 1307 1305 1305
query25 570 495 432 432
query26 1315 317 172 172
query27 2709 563 344 344
query28 4560 1972 1928 1928
query29 975 652 521 521
query30 306 239 202 202
query31 1114 1060 949 949
query32 86 80 74 74
query33 545 360 312 312
query34 1191 1164 632 632
query35 772 781 676 676
query36 1354 1321 1164 1164
query37 157 104 95 95
query38 3223 3102 3034 3034
query39 940 914 890 890
query39_1 872 877 898 877
query40 232 149 131 131
query41 74 74 70 70
query42 111 112 110 110
query43 328 335 294 294
query44
query45 212 206 200 200
query46 1051 1180 713 713
query47 2366 2365 2251 2251
query48 410 410 317 317
query49 661 506 403 403
query50 957 358 249 249
query51 4320 4323 4264 4264
query52 107 106 98 98
query53 260 280 203 203
query54 326 298 267 267
query55 96 96 87 87
query56 310 324 312 312
query57 1428 1438 1343 1343
query58 314 277 272 272
query59 1576 1651 1441 1441
query60 323 324 304 304
query61 157 154 150 150
query62 683 614 548 548
query63 243 203 206 203
query64 2409 812 634 634
query65
query66 1726 464 344 344
query67 30017 29935 29730 29730
query68
query69 470 345 305 305
query70 1000 1012 1002 1002
query71 313 270 267 267
query72 3005 2730 2396 2396
query73 846 724 420 420
query74 5050 4859 4718 4718
query75 2669 2585 2273 2273
query76 2299 1143 742 742
query77 385 400 331 331
query78 12097 12097 11596 11596
query79 1423 1065 737 737
query80 710 553 442 442
query81 456 282 238 238
query82 1399 157 121 121
query83 334 286 253 253
query84 310 181 113 113
query85 877 534 454 454
query86 389 334 321 321
query87 3383 3352 3235 3235
query88 3501 2680 2657 2657
query89 451 379 339 339
query90 1891 179 176 176
query91 178 165 149 149
query92 84 81 71 71
query93 1471 1388 960 960
query94 559 343 317 317
query95 685 494 369 369
query96 1006 785 342 342
query97 2709 2717 2568 2568
query98 235 229 248 229
query99 1112 1107 975 975
Total cold run time: 252774 ms
Total hot run time: 169122 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant decimal precision [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4470912196
<details>
<summary>TPC-H: <b>Total hot run time: 31044 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3dd972bebfba3e772374972a1829da943aea8361, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17632 3816 3797 3797
q2 q3 10760 1415 870 870
q4 4687 471 349 349
q5 7633 2286 2093 2093
q6 375 183 139 139
q7 959 769 620 620
q8 9538 1702 1534 1534
q9 7081 4993 4947 4947
q10 6464 2086 1789 1789
q11 440 283 247 247
q12 698 431 298 298
q13 18180 3458 2769 2769
q14 263 260 234 234
q15 q16 821 779 708 708
q17 1022 959 974 959
q18 6979 5791 5533 5533
q19 1201 1228 1092 1092
q20 519 406 264 264
q21 5730 2552 2493 2493
q22 438 363 309 309
Total cold run time: 101420 ms
Total hot run time: 31044 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4202 4107 4117 4107
q2 q3 4501 5023 4324 4324
q4 2100 2213 1407 1407
q5 4533 4325 4474 4325
q6 281 201 151 151
q7 2063 1769 1673 1673
q8 2489 2108 2194 2108
q9 7835 7855 7709 7709
q10 4566 4493 4061 4061
q11 808 421 380 380
q12 716 738 505 505
q13 3324 3787 2922 2922
q14 284 292 276 276
q15 q16 736 749 671 671
q17 1358 1312 1370 1312
q18 7957 7401 6942 6942
q19 1117 1105 1096 1096
q20 2237 2217 1916 1916
q21 5310 4581 4492 4492
q22 538 457 414 414
Total cold run time: 56955 ms
Total hot run time: 50791 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant decimal precision [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4470970037
<details>
<summary>TPC-H: <b>Total hot run time: 31302 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 47d2d8e39a7486d4aec3c55d15f426a812f9df4c, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17854 4033 3981 3981
q2 q3 10772 1424 809 809
q4 4679 477 351 351
q5 7601 2237 2134 2134
q6 263 192 144 144
q7 961 788 633 633
q8 9358 1775 1650 1650
q9 6838 4933 4916 4916
q10 6435 2099 1780 1780
q11 432 278 243 243
q12 718 426 296 296
q13 18213 3397 2764 2764
q14 265 256 240 240
q15 q16 817 767 703 703
q17 984 974 958 958
q18 6877 5774 5611 5611
q19 1239 1252 1146 1146
q20 512 412 258 258
q21 5767 2577 2379 2379
q22 427 363 306 306
Total cold run time: 101012 ms
Total hot run time: 31302 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4370 4232 4213 4213
q2 q3 4534 4939 4378 4378
q4 2092 2207 1387 1387
q5 4398 4308 4630 4308
q6 264 206 154 154
q7 1991 1886 1584 1584
q8 2473 2162 2137 2137
q9 7723 7882 7763 7763
q10 4537 4477 4188 4188
q11 584 413 377 377
q12 713 744 542 542
q13 3334 3630 3011 3011
q14 316 321 311 311
q15 q16 730 724 673 673
q17 1385 1335 1384 1335
q18 8085 7307 6752 6752
q19 1161 1090 1100 1090
q20 2212 2217 1936 1936
q21 5315 4642 4460 4460
q22 536 478 430 430
Total cold run time: 56753 ms
Total hot run time: 51029 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant decimal precision [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4471250567
<details>
<summary>TPC-H: <b>Total hot run time: 31380 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6d77c2e151db0215a94add25f0ed9e0300fa3649, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17667 4058 4099 4058
q2 q3 10791 1420 804 804
q4 4679 466 352 352
q5 7557 2267 2148 2148
q6 378 175 139 139
q7 985 764 679 679
q8 9374 1690 1549 1549
q9 6793 4930 4934 4930
q10 6463 2127 1808 1808
q11 435 269 244 244
q12 690 433 290 290
q13 18213 3302 2805 2805
q14 275 255 243 243
q15 q16 828 768 717 717
q17 1023 974 934 934
q18 6914 5870 5656 5656
q19 1300 1228 1066 1066
q20 524 422 265 265
q21 5795 2634 2385 2385
q22 435 354 308 308
Total cold run time: 101119 ms
Total hot run time: 31380 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4281 4214 4305 4214
q2 q3 4532 4940 4301 4301
q4 2117 2176 1401 1401
q5 4387 4299 4632 4299
q6 247 200 143 143
q7 2012 1806 1696 1696
q8 2494 2122 2150 2122
q9 7786 7855 7792 7792
q10 4539 4510 4122 4122
q11 636 422 386 386
q12 707 749 512 512
q13 3309 3654 2958 2958
q14 300 297 291 291
q15 q16 727 723 661 661
q17 1425 1351 1368 1351
q18 7999 7528 6945 6945
q19 1151 1099 1108 1099
q20 2212 2215 1936 1936
q21 5346 4697 4491 4491
q22 522 472 406 406
Total cold run time: 56729 ms
Total hot run time: 51126 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant decimal precision [doris]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4470181486
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant decimal precision [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4471465480
# BE UT Coverage Report
Increment line coverage `72.04% (304/422)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/95ccafea2253112ebae2ad285d71df8be7689d9b_95ccafea2253112ebae2ad285d71df8be7689d9b/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/95ccafea2253112ebae2ad285d71df8be7689d9b_95ccafea2253112ebae2ad285d71df8be7689d9b/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 53.51% (20659/38607) |
| Line Coverage | 37.17% (195318/525439) |
| Region Coverage | 33.56% (152947/455703) |
| Branch Coverage | 34.60% (66716/192829) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant predefined decimal precision [doris]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4474456881
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant predefined decimal precision [doris]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4475844996
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant decimal precision [doris]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4470179390
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant decimal precision [doris]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4470319139
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant decimal precision [doris]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4470763987
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant decimal precision [doris]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4471037436
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant decimal precision [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4471499463
<details>
<summary>TPC-H: <b>Total hot run time: 30776 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 45d9060c080f192ecb190462a4b01b6485daad2a, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17637 3846 3878 3846
q2 q3 10745 1410 812 812
q4 4688 475 352 352
q5 7634 2237 2071 2071
q6 240 177 138 138
q7 940 776 639 639
q8 9392 1666 1578 1578
q9 6581 4902 4886 4886
q10 6432 2123 1771 1771
q11 433 271 248 248
q12 693 422 291 291
q13 18247 3329 2784 2784
q14 264 254 232 232
q15 q16 823 771 703 703
q17 972 940 935 935
q18 6815 5835 5529 5529
q19 1238 1165 1089 1089
q20 505 392 269 269
q21 5620 2596 2303 2303
q22 426 359 300 300
Total cold run time: 100325 ms
Total hot run time: 30776 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4247 4107 4074 4074
q2 q3 4542 4868 4334 4334
q4 2113 2171 1372 1372
q5 4366 4252 4227 4227
q6 222 174 133 133
q7 2061 1857 1706 1706
q8 2536 2084 2035 2035
q9 7781 7743 7732 7732
q10 4530 4478 4088 4088
q11 617 427 373 373
q12 711 736 532 532
q13 3480 3639 3034 3034
q14 280 311 291 291
q15 q16 747 739 675 675
q17 1373 1335 1302 1302
q18 7950 7343 7098 7098
q19 1099 1077 1125 1077
q20 2229 2222 1941 1941
q21 5280 4647 4455 4455
q22 549 454 406 406
Total cold run time: 56713 ms
Total hot run time: 50885 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant predefined decimal precision [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #63322:
URL: https://github.com/apache/doris/pull/63322#discussion_r3256376020
##########
be/src/util/json/json_parser.cpp:
##########
@@ -84,8 +89,29 @@ void JSONDataParser<ParserImpl>::traverse(const Element& element, ParseContext&
// we should set has_nested_in_flatten to false when traverse array finished for next array otherwise it will be true for next array
ctx.has_nested_in_flatten = false;
} else {
- appendValueIfNotDuplicate(ctx, ctx.builder.get_parts(), getValueAsField(element));
+ appendValueIfNotDuplicate(ctx, ctx.builder.get_parts(),
+ getValueAsField(element, shouldPreserveNumberAsString(ctx)));
+ }
Review Comment:
The raw-number decision is only used on the scalar `Field` path. When an array contains objects and `deprecated_enable_flatten_nested` is false, the branch above serializes the whole array with `traverseArrayAsJsonb()` instead of recursing to leaf paths; `writeValueAsJsonb()` still writes floating-point numbers via `writer.writeDouble(element.getDouble())`. A predefined decimal path inside an array of objects, for example `items.price` with JSON `{"items":[{"price":999999999999999999999999999.999999999}]}`, can therefore be rounded before the storage-layer nested expansion/materialization casts it to decimal. Please preserve raw numeric text in the JSONB serialization path for configured decimal typed paths as well, or avoid routing those typed paths through lossy JSONB serialization, and add a regression with a predefined decimal inside an array of objects.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant predefined decimal precision [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #63322:
URL: https://github.com/apache/doris/pull/63322#discussion_r3256580700
##########
be/src/exec/common/variant_util.cpp:
##########
@@ -104,6 +104,44 @@
namespace doris::variant_util {
+static bool is_decimal_typed_path_column(const TabletColumn& column) {
+ if (column.is_array_type()) {
+ CHECK_EQ(column.get_sub_columns().size(), 1);
+ return is_decimal_typed_path_column(*column.get_sub_columns()[0]);
+ }
+ return is_decimal(TabletColumn::get_primitive_type_by_field_type(column.type()));
+}
+
+static void configure_decimal_number_preserve_paths(const TabletColumn& column,
Review Comment:
This helper is only applied from `parse_and_materialize_variant_columns()`, after the input block already contains a scalar-root Variant that still needs materialization. A direct JSON stream load into a Variant column with an object value, for example `{"id":2,"v":{"number_1":999999999999999999999999999.999999999}}`, goes through `DataTypeVariantSerDe::deserialize_one_cell_from_json()`, where `ParseConfig` is created without the tablet schema's decimal typed paths. That parse builds a non-scalar `ColumnVariant`, so the later writer path hits `if (!var.is_scalar_variant()) continue;` and never reaches this configured materialization path. The new regression only stream-loads `v` as an escaped JSON string, which exercises scalar-root materialization, not the direct object path. Please propagate the schema-derived decimal preserve paths to direct Variant JSON deserialization as well, or keep object-valued stream-loads as raw scalar JSON until this materialization step, and add a reg
ression where the stream-load file contains `v` as an object rather than a string.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant predefined decimal precision [doris]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4474825492
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant predefined decimal precision [doris]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4474824733
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant decimal precision [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #63322:
URL: https://github.com/apache/doris/pull/63322#discussion_r3254468041
##########
be/src/util/json/json_parser.h:
##########
@@ -39,8 +39,47 @@
namespace doris {
+inline bool should_preserve_json_number_as_string(std::string_view raw_number) {
+ size_t significant_digits = 0;
+ size_t fractional_digits = 0;
+ bool seen_nonzero_digit = false;
+ bool seen_dot = false;
+ bool seen_exponent = false;
+
+ for (size_t i = 0; i < raw_number.size(); ++i) {
+ const char ch = raw_number[i];
+ if (ch == 'e' || ch == 'E') {
+ seen_exponent = true;
+ if (i + 1 < raw_number.size() &&
+ (raw_number[i + 1] == '-' || raw_number[i + 1] == '+')) {
+ ++i;
+ }
+ continue;
+ }
+ if (seen_exponent) {
+ continue;
+ }
+ if (ch == '.') {
+ seen_dot = true;
+ continue;
+ }
+ if (ch < '0' || ch > '9') {
+ continue;
+ }
+ if (seen_dot) {
+ ++fractional_digits;
+ }
+ if (ch != '0' || seen_nonzero_digit) {
+ seen_nonzero_digit = true;
+ ++significant_digits;
+ }
+ }
+
+ return significant_digits > 15 || fractional_digits >= 10;
+}
Review Comment:
This heuristic still lets typed decimal paths lose precision for scientific-notation tokens because it stops counting once it sees `e`/`E` and never accounts for the exponent magnitude. For example, with a `decimalv3(38,10)` typed path, `9.999999999e27` has only 10 mantissa significant digits and 9 fractional digits before the exponent, so this returns false; `getValueAsField()` then stores a `TYPE_DOUBLE`, and the later decimal cast uses the rounded double instead of the original JSON token. Please preserve the raw token for exponent-form decimal typed-path numbers as well, or preserve raw numeric text for all double typed-path values before casting to decimal.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant decimal precision [doris]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4470612315
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant decimal precision [doris]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4470508551
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant decimal precision [doris]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4471451200
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant predefined decimal precision [doris]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4474525624
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant decimal precision [doris]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4471220496
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant decimal precision [doris]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4471350370
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant predefined decimal precision [doris]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4473927544
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant predefined decimal precision [doris]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4474456379
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant predefined decimal precision [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4477200478
<details>
<summary>TPC-DS: <b>Total hot run time: 168919 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1f40adb4feed235b8c3c5c6e215682a37a4efb68, data reload: false
query5 4333 663 516 516
query6 330 215 210 210
query7 4250 569 304 304
query8 330 232 213 213
query9 8817 3977 3979 3977
query10 448 335 294 294
query11 5772 2361 2245 2245
query12 181 129 127 127
query13 1302 618 447 447
query14 5982 5378 5067 5067
query14_1 4359 4340 4366 4340
query15 228 202 188 188
query16 1044 461 431 431
query17 1172 724 571 571
query18 2682 476 348 348
query19 210 198 160 160
query20 137 130 129 129
query21 225 145 118 118
query22 13671 13626 13449 13449
query23 17220 16342 16020 16020
query23_1 16249 16184 16150 16150
query24 7399 1739 1275 1275
query24_1 1292 1281 1299 1281
query25 541 472 416 416
query26 1300 320 176 176
query27 2684 587 354 354
query28 4417 2017 1964 1964
query29 987 611 492 492
query30 312 238 200 200
query31 1101 1070 930 930
query32 90 76 74 74
query33 540 349 283 283
query34 1179 1121 666 666
query35 747 760 676 676
query36 1329 1320 1150 1150
query37 156 111 105 105
query38 3215 3106 3053 3053
query39 915 943 901 901
query39_1 903 879 888 879
query40 222 144 126 126
query41 97 63 61 61
query42 110 107 108 107
query43 322 332 288 288
query44
query45 210 205 196 196
query46 1074 1204 710 710
query47 2278 2398 2188 2188
query48 394 396 291 291
query49 637 494 383 383
query50 977 348 236 236
query51 4293 4279 4193 4193
query52 106 104 94 94
query53 263 286 207 207
query54 302 286 261 261
query55 96 88 85 85
query56 309 322 303 303
query57 1421 1397 1339 1339
query58 294 269 267 267
query59 1549 1625 1424 1424
query60 322 333 306 306
query61 160 157 156 156
query62 675 640 562 562
query63 241 201 205 201
query64 2426 810 630 630
query65
query66 1701 488 354 354
query67 29481 29980 29332 29332
query68
query69 465 343 301 301
query70 1021 1003 1029 1003
query71 303 278 268 268
query72 2971 2688 2386 2386
query73 871 762 424 424
query74 5070 4878 4738 4738
query75 2670 2588 2256 2256
query76 2317 1127 733 733
query77 395 408 334 334
query78 12116 12099 11681 11681
query79 1262 1035 747 747
query80 594 536 444 444
query81 452 279 254 254
query82 238 159 125 125
query83 282 288 255 255
query84 272 143 117 117
query85 955 623 538 538
query86 361 333 327 327
query87 3392 3396 3203 3203
query88 3512 2671 2668 2668
query89 431 404 343 343
query90 2243 182 181 181
query91 195 189 157 157
query92 80 83 77 77
query93 1458 1407 926 926
query94 544 317 331 317
query95 700 414 452 414
query96 1082 769 355 355
query97 2711 2716 2612 2612
query98 240 239 231 231
query99 1143 1101 990 990
Total cold run time: 251255 ms
Total hot run time: 168919 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant decimal precision [doris]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4470881569
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant decimal precision [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4470904349
<details>
<summary>TPC-H: <b>Total hot run time: 31177 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9a398bd168d5dc2f4047719bccd4b533f710ea0b, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17633 3973 3914 3914
q2 q3 10790 1404 820 820
q4 4686 475 351 351
q5 7549 2282 2136 2136
q6 235 182 143 143
q7 941 789 614 614
q8 9367 1660 1637 1637
q9 5134 4935 4983 4935
q10 6389 2096 1801 1801
q11 448 274 249 249
q12 631 431 299 299
q13 18109 3358 2765 2765
q14 261 257 234 234
q15 q16 815 772 712 712
q17 928 896 953 896
q18 7106 5812 5546 5546
q19 1285 1347 1041 1041
q20 531 404 266 266
q21 6161 2744 2518 2518
q22 438 369 300 300
Total cold run time: 99437 ms
Total hot run time: 31177 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4198 4129 4193 4129
q2 q3 4575 4926 4316 4316
q4 2130 2225 1397 1397
q5 4429 4310 4335 4310
q6 239 188 140 140
q7 2281 1985 1717 1717
q8 2606 2323 2330 2323
q9 8096 7773 7734 7734
q10 4569 4482 4072 4072
q11 582 427 424 424
q12 748 741 555 555
q13 3255 3677 2948 2948
q14 307 304 286 286
q15 q16 709 753 676 676
q17 1398 1381 1440 1381
q18 8287 7407 7510 7407
q19 1192 1146 1095 1095
q20 2223 2218 1914 1914
q21 5413 4755 4706 4706
q22 538 486 436 436
Total cold run time: 57775 ms
Total hot run time: 51966 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant decimal precision [doris]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4470319367
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant decimal precision [doris]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4470808046
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant decimal precision [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #63322:
URL: https://github.com/apache/doris/pull/63322#discussion_r3255002040
##########
be/src/util/json/simd_json_parser.h:
##########
@@ -38,143 +72,400 @@ class SimdJSONParser {
class Element {
public:
ALWAYS_INLINE Element() {} /// NOLINT
- ALWAYS_INLINE Element(const simdjson::dom::element& element_)
- : element(element_) {} /// NOLINT
+ ALWAYS_INLINE explicit Element(const simdjson::dom::element& element_)
+ : dom_element(element_) {} /// NOLINT
+ ALWAYS_INLINE explicit Element(const Node* node_) : node(node_) {} /// NOLINT
ALWAYS_INLINE bool isInt64() const {
- return element.type() == simdjson::dom::element_type::INT64;
+ return node ? node->type == Node::Type::INT64
+ : dom_element.type() == simdjson::dom::element_type::INT64;
}
ALWAYS_INLINE bool isUInt64() const {
- return element.type() == simdjson::dom::element_type::UINT64;
+ return node ? node->type == Node::Type::UINT64
+ : dom_element.type() == simdjson::dom::element_type::UINT64;
}
+ ALWAYS_INLINE bool isInt128() const { return node && node->type == Node::Type::INT128; }
ALWAYS_INLINE bool isDouble() const {
- return element.type() == simdjson::dom::element_type::DOUBLE;
+ return node ? node->type == Node::Type::DOUBLE
+ : dom_element.type() == simdjson::dom::element_type::DOUBLE;
}
ALWAYS_INLINE bool isString() const {
- return element.type() == simdjson::dom::element_type::STRING;
+ return node ? node->type == Node::Type::STRING
+ : dom_element.type() == simdjson::dom::element_type::STRING;
}
ALWAYS_INLINE bool isArray() const {
- return element.type() == simdjson::dom::element_type::ARRAY;
+ return node ? node->type == Node::Type::ARRAY
+ : dom_element.type() == simdjson::dom::element_type::ARRAY;
}
ALWAYS_INLINE bool isObject() const {
- return element.type() == simdjson::dom::element_type::OBJECT;
+ return node ? node->type == Node::Type::OBJECT
+ : dom_element.type() == simdjson::dom::element_type::OBJECT;
}
ALWAYS_INLINE bool isBool() const {
- return element.type() == simdjson::dom::element_type::BOOLEAN;
+ return node ? node->type == Node::Type::BOOL
+ : dom_element.type() == simdjson::dom::element_type::BOOLEAN;
}
ALWAYS_INLINE bool isNull() const {
- return element.type() == simdjson::dom::element_type::NULL_VALUE;
+ return node ? node->type == Node::Type::NULL_VALUE
+ : dom_element.type() == simdjson::dom::element_type::NULL_VALUE;
+ }
+ ALWAYS_INLINE Int64 getInt64() const {
+ return node ? node->int64_value : dom_element.get_int64().value_unsafe();
+ }
+ ALWAYS_INLINE double getDouble() const {
+ return node ? node->double_value : dom_element.get_double().value_unsafe();
+ }
+ ALWAYS_INLINE bool getBool() const {
+ return node ? node->bool_value : dom_element.get_bool().value_unsafe();
}
- ALWAYS_INLINE Int64 getInt64() const { return element.get_int64().value_unsafe(); }
- ALWAYS_INLINE double getDouble() const { return element.get_double().value_unsafe(); }
- ALWAYS_INLINE bool getBool() const { return element.get_bool().value_unsafe(); }
ALWAYS_INLINE std::string_view getString() const {
- return element.get_string().value_unsafe();
+ return node ? std::string_view(node->string_value)
+ : dom_element.get_string().value_unsafe();
+ }
+ ALWAYS_INLINE UInt64 getUInt64() const {
+ return node ? node->uint64_value : dom_element.get_uint64().value_unsafe();
+ }
+ ALWAYS_INLINE Int128 getInt128() const {
+ assert(node != nullptr);
+ return node->int128_value;
+ }
+ ALWAYS_INLINE std::string_view getRawNumber() const {
+ assert(node != nullptr);
+ return node->raw_number;
}
- ALWAYS_INLINE UInt64 getUInt64() const { return element.get_uint64().value_unsafe(); }
ALWAYS_INLINE Array getArray() const;
ALWAYS_INLINE Object getObject() const;
private:
- simdjson::dom::element element;
+ simdjson::dom::element dom_element;
+ const Node* node = nullptr;
};
/// References an array in a JSON document.
class Array {
public:
class Iterator {
public:
- ALWAYS_INLINE Iterator(const simdjson::dom::array::iterator& it_)
- : it(it_) {} /// NOLINT
- ALWAYS_INLINE Element operator*() const { return *it; }
+ using DomIterator = simdjson::dom::array::iterator;
+ using NodeIterator = std::vector<Node>::const_iterator;
+ ALWAYS_INLINE explicit Iterator(const DomIterator& it_) : it(it_) {} /// NOLINT
+ ALWAYS_INLINE explicit Iterator(NodeIterator it_) : it(it_) {} /// NOLINT
+ ALWAYS_INLINE Element operator*() const {
+ if (const auto* node_it = std::get_if<NodeIterator>(&it)) {
+ return Element(&**node_it);
+ }
+ return Element(*std::get<DomIterator>(it));
+ }
ALWAYS_INLINE Iterator& operator++() {
- ++it;
+ if (auto* node_it = std::get_if<NodeIterator>(&it)) {
+ ++(*node_it);
+ } else {
+ ++std::get<DomIterator>(it);
+ }
return *this;
}
ALWAYS_INLINE friend bool operator!=(const Iterator& left, const Iterator& right) {
- return left.it != right.it;
+ if (const auto* left_node_it = std::get_if<NodeIterator>(&left.it)) {
+ return *left_node_it != std::get<NodeIterator>(right.it);
+ }
+ return std::get<DomIterator>(left.it) != std::get<DomIterator>(right.it);
}
private:
- simdjson::dom::array::iterator it;
+ std::variant<DomIterator, NodeIterator> it;
};
- ALWAYS_INLINE Array(const simdjson::dom::array& array_) : array(array_) {} /// NOLINT
- ALWAYS_INLINE Iterator begin() const { return array.begin(); }
- ALWAYS_INLINE Iterator end() const { return array.end(); }
- ALWAYS_INLINE size_t size() const { return array.size(); }
+ ALWAYS_INLINE explicit Array(const simdjson::dom::array& array_)
+ : dom_array(array_) {} /// NOLINT
+ ALWAYS_INLINE explicit Array(const std::vector<Node>* array_) : array(array_) {} /// NOLINT
+ ALWAYS_INLINE Iterator begin() const {
+ return array ? Iterator(array->begin()) : Iterator(dom_array.begin());
+ }
+ ALWAYS_INLINE Iterator end() const {
+ return array ? Iterator(array->end()) : Iterator(dom_array.end());
+ }
+ ALWAYS_INLINE size_t size() const { return array ? array->size() : dom_array.size(); }
ALWAYS_INLINE Element operator[](size_t index) const {
assert(index < size());
- return array.at(index).value_unsafe();
+ return array ? Element(&(*array)[index]) : Element(dom_array.at(index).value_unsafe());
}
private:
- simdjson::dom::array array;
+ simdjson::dom::array dom_array;
+ const std::vector<Node>* array = nullptr;
};
using KeyValuePair = std::pair<std::string_view, Element>;
/// References an object in a JSON document.
class Object {
public:
class Iterator {
public:
- ALWAYS_INLINE Iterator(const simdjson::dom::object::iterator& it_)
- : it(it_) {} /// NOLINT
+ using DomIterator = simdjson::dom::object::iterator;
+ ALWAYS_INLINE explicit Iterator(const std::vector<std::string>* keys_,
+ const std::vector<Node>* values_, size_t index_)
+ : it(index_), keys(keys_), values(values_) {} /// NOLINT
+ ALWAYS_INLINE explicit Iterator(const DomIterator& it_) : it(it_) {} /// NOLINT
ALWAYS_INLINE KeyValuePair operator*() const {
- const auto& res = *it;
- return {res.key, res.value};
+ if (const auto* index = std::get_if<size_t>(&it)) {
+ return {(*keys)[*index], Element(&(*values)[*index])};
+ }
+ const auto& res = *std::get<DomIterator>(it);
+ return {res.key, Element(res.value)};
}
ALWAYS_INLINE Iterator& operator++() {
- ++it;
+ if (auto* index = std::get_if<size_t>(&it)) {
+ ++(*index);
+ } else {
+ ++std::get<DomIterator>(it);
+ }
return *this;
}
ALWAYS_INLINE Iterator operator++(int) {
auto res = *this;
- ++it;
+ ++(*this);
return res;
} /// NOLINT
ALWAYS_INLINE friend bool operator!=(const Iterator& left, const Iterator& right) {
- return left.it != right.it;
+ if (const auto* left_index = std::get_if<size_t>(&left.it)) {
+ return *left_index != std::get<size_t>(right.it);
+ }
+ return std::get<DomIterator>(left.it) != std::get<DomIterator>(right.it);
}
ALWAYS_INLINE friend bool operator==(const Iterator& left, const Iterator& right) {
return !(left != right);
}
private:
- simdjson::dom::object::iterator it;
+ std::variant<DomIterator, size_t> it;
+ const std::vector<std::string>* keys = nullptr;
+ const std::vector<Node>* values = nullptr;
};
- ALWAYS_INLINE Object(const simdjson::dom::object& object_) : object(object_) {} /// NOLINT
- ALWAYS_INLINE Iterator begin() const { return object.begin(); }
- ALWAYS_INLINE Iterator end() const { return object.end(); }
- ALWAYS_INLINE size_t size() const { return object.size(); }
+ ALWAYS_INLINE explicit Object(const simdjson::dom::object& object_)
+ : dom_object(object_) {} /// NOLINT
+ ALWAYS_INLINE explicit Object(const std::vector<std::string>* keys_,
+ const std::vector<Node>* values_)
+ : keys(keys_), values(values_) {} /// NOLINT
+ ALWAYS_INLINE Iterator begin() const {
+ return values ? Iterator(keys, values, 0) : Iterator(dom_object.begin());
+ }
+ ALWAYS_INLINE Iterator end() const {
+ return values ? Iterator(keys, values, size()) : Iterator(dom_object.end());
+ }
+ ALWAYS_INLINE size_t size() const { return values ? values->size() : dom_object.size(); }
/// Optional: Provides access to an object's element by index.
KeyValuePair operator[](size_t index) const {
assert(index < size());
- auto it = object.begin();
+ if (values) {
+ return {(*keys)[index], Element(&(*values)[index])};
+ }
+ auto it = dom_object.begin();
while (index--) {
++it;
}
const auto& res = *it;
- return {res.key, res.value};
+ return {res.key, Element(res.value)};
}
private:
- simdjson::dom::object object;
+ simdjson::dom::object dom_object;
+ const std::vector<std::string>* keys = nullptr;
+ const std::vector<Node>* values = nullptr;
};
/// Parses a JSON document, returns the reference to its root element if succeeded.
- bool parse(const char* data, size_t size, Element& result) {
- auto document = parser.parse(data, size);
- if (document.error()) {
+ bool parse(const char* data, size_t size, Element& result, bool preserve_raw_numbers = false) {
+ if (!preserve_raw_numbers) {
+ auto document = dom_parser.parse(data, size);
+ if (document.error()) {
+ return false;
+ }
+ result = Element(document.value_unsafe());
+ return true;
+ }
+
+ return parse_ondemand(data, size, result);
+ }
+
+private:
+ bool parse_ondemand(const char* data, size_t size, Element& result) {
+ simdjson::padded_string padded_json(data, size);
+ simdjson::ondemand::document document;
+ auto error = ondemand_parser.iterate(padded_json).get(document);
+ if (error) {
+ return false;
+ }
+ root = Node();
+ if (!build_node(document, &root)) {
return false;
}
- result = document.value_unsafe();
+ result = Element(&root);
return true;
}
-private:
- simdjson::dom::parser parser;
+ static std::string_view trim_raw_number(std::string_view raw_number) {
+ auto is_space = [](char ch) { return ch == ' ' || ch == '\t' || ch == '\n' || ch == '\r'; };
+ while (!raw_number.empty() && is_space(raw_number.front())) {
+ raw_number.remove_prefix(1);
+ }
+ while (!raw_number.empty() && is_space(raw_number.back())) {
+ raw_number.remove_suffix(1);
+ }
+ return raw_number;
+ }
+
+ template <typename RawNumber>
+ static bool assign_raw_number(RawNumber&& raw_number, std::string* out) {
+ if constexpr (std::is_same_v<std::decay_t<RawNumber>, std::string_view>) {
+ *out = std::string(trim_raw_number(raw_number));
+ return true;
+ } else {
+ std::string_view raw_number_view;
+ auto error = std::move(raw_number).get(raw_number_view);
+ if (error) {
+ return false;
+ }
+ *out = std::string(trim_raw_number(raw_number_view));
+ return true;
+ }
+ }
+
+ template <typename Value>
+ bool build_array_node(Value& value, Node* out) {
+ simdjson::ondemand::array array;
+ auto error = value.get_array().get(array);
+ if (error) {
+ return false;
+ }
+ out->type = Node::Type::ARRAY;
+ for (auto element_result : array) {
+ simdjson::ondemand::value element;
+ error = std::move(element_result).get(element);
+ if (error) {
+ return false;
+ }
+ Node element_node;
+ if (!build_node(element, &element_node)) {
+ return false;
+ }
+ out->array_values.push_back(std::move(element_node));
+ }
+ return true;
+ }
+
+ template <typename Value>
+ bool build_object_node(Value& value, Node* out) {
+ simdjson::ondemand::object object;
+ auto error = value.get_object().get(object);
+ if (error) {
+ return false;
+ }
+ out->type = Node::Type::OBJECT;
+ for (auto field_result : object) {
+ simdjson::ondemand::field field;
+ error = std::move(field_result).get(field);
+ if (error) {
+ return false;
+ }
+ std::string_view key;
+ error = field.unescaped_key().get(key);
+ if (error) {
+ return false;
+ }
+ std::string key_copy(key);
+ simdjson::ondemand::value field_value = field.value();
+ Node field_node;
+ if (!build_node(field_value, &field_node)) {
+ return false;
+ }
+ out->object_keys.push_back(std::move(key_copy));
+ out->object_values.push_back(std::move(field_node));
+ }
+ return true;
+ }
+
+ template <typename Value>
+ bool build_number_node(Value& value, Node* out) {
+ simdjson::ondemand::number_type number_type;
+ auto error = value.get_number_type().get(number_type);
+ if (error) {
+ return false;
+ }
+ switch (number_type) {
+ case simdjson::ondemand::number_type::signed_integer:
+ out->type = Node::Type::INT64;
+ error = value.get_int64().get(out->int64_value);
+ return !error;
+ case simdjson::ondemand::number_type::unsigned_integer:
+ out->type = Node::Type::UINT64;
+ error = value.get_uint64().get(out->uint64_value);
+ return !error;
+ case simdjson::ondemand::number_type::floating_point_number:
+ if (!assign_raw_number(value.raw_json_token(), &out->raw_number)) {
+ return false;
+ }
+ out->type = Node::Type::DOUBLE;
+ error = value.get_double().get(out->double_value);
+ return !error;
+ case simdjson::ondemand::number_type::big_integer: {
+ if (!assign_raw_number(value.raw_json_token(), &out->raw_number)) {
+ return false;
+ }
+ out->type = Node::Type::INT128;
+ StringParser::ParseResult parse_result;
+ out->int128_value = StringParser::string_to_int<Int128>(
+ out->raw_number.data(), out->raw_number.size(), &parse_result);
+ return parse_result == StringParser::PARSE_SUCCESS;
+ }
Review Comment:
This `big_integer` branch still requires every raw-preservation parse to fit the token into `Int128` before `JSONDataParser` can check whether the current path is a typed decimal path. A Variant path declared as `decimalv3(76,0)`/Decimal256 can validly contain an integer larger than `Int128`, for example `340282366920938463463374607431768211456`; with a schema typed path this should be preserved as raw text and cast to Decimal256, but `string_to_int<Int128>()` fails here, `build_node()` returns false, and the whole JSON is treated as parse failure/fallback instead. Please keep the raw token for `big_integer` nodes without requiring an Int128 conversion on paths that may be decimal-preserved, or defer the Int128 conversion until the consumer actually needs a LargeInt, and add a Decimal256 integer regression case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant decimal precision [doris]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4470611937
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant decimal precision [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4471275786
<details>
<summary>TPC-DS: <b>Total hot run time: 170186 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6d77c2e151db0215a94add25f0ed9e0300fa3649, data reload: false
query5 4315 663 516 516
query6 331 221 197 197
query7 4278 553 296 296
query8 323 230 225 225
query9 8832 4064 4025 4025
query10 456 344 288 288
query11 5863 2411 2205 2205
query12 175 127 121 121
query13 1330 640 432 432
query14 6024 5415 5108 5108
query14_1 4414 4396 4377 4377
query15 210 205 176 176
query16 1014 448 417 417
query17 1025 706 590 590
query18 2484 495 348 348
query19 236 205 161 161
query20 135 132 129 129
query21 220 146 117 117
query22 13773 13494 13421 13421
query23 17140 16530 16119 16119
query23_1 16389 16401 16291 16291
query24 7614 1746 1280 1280
query24_1 1326 1296 1264 1264
query25 559 473 428 428
query26 1368 319 175 175
query27 2641 560 343 343
query28 4458 1921 1923 1921
query29 1006 621 490 490
query30 328 241 198 198
query31 1102 1055 959 959
query32 87 76 74 74
query33 540 348 294 294
query34 1158 1222 642 642
query35 765 776 691 691
query36 1359 1333 1152 1152
query37 157 106 92 92
query38 3250 3150 3074 3074
query39 950 941 894 894
query39_1 889 882 903 882
query40 252 158 130 130
query41 72 69 70 69
query42 117 113 111 111
query43 340 335 304 304
query44
query45 217 204 208 204
query46 1102 1185 737 737
query47 2337 2330 2156 2156
query48 411 428 301 301
query49 657 524 402 402
query50 982 363 251 251
query51 4446 4314 4212 4212
query52 107 106 96 96
query53 261 285 209 209
query54 331 286 268 268
query55 94 92 88 88
query56 324 325 325 325
query57 1419 1383 1267 1267
query58 311 285 274 274
query59 1607 1695 1486 1486
query60 343 333 322 322
query61 183 184 182 182
query62 670 632 554 554
query63 243 205 214 205
query64 2500 882 693 693
query65
query66 1738 493 362 362
query67 30381 30091 30122 30091
query68
query69 485 352 307 307
query70 1016 986 1040 986
query71 320 280 281 280
query72 3047 2657 2420 2420
query73 860 811 436 436
query74 5086 4925 4737 4737
query75 2700 2591 2261 2261
query76 2284 1149 764 764
query77 402 419 333 333
query78 12160 12161 11651 11651
query79 1479 1022 699 699
query80 1242 557 496 496
query81 510 279 244 244
query82 1355 158 122 122
query83 359 277 261 261
query84 302 138 110 110
query85 937 539 455 455
query86 452 351 320 320
query87 3438 3379 3237 3237
query88 3593 2649 2632 2632
query89 458 382 336 336
query90 1796 181 185 181
query91 180 177 142 142
query92 82 88 77 77
query93 1508 1417 879 879
query94 681 329 308 308
query95 689 395 434 395
query96 985 793 353 353
query97 2674 2671 2575 2575
query98 239 230 228 228
query99 1134 1089 986 986
Total cold run time: 255506 ms
Total hot run time: 170186 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant decimal precision [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4470493827
<details>
<summary>TPC-H: <b>Total hot run time: 31205 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 832ca14461bc0faf4fbf9b0d2a91c7df1111f393, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17647 3855 3817 3817
q2 q3 10758 1360 803 803
q4 4705 479 344 344
q5 8069 2303 2126 2126
q6 386 176 138 138
q7 955 770 647 647
q8 9675 1690 1594 1594
q9 7088 4964 4934 4934
q10 6460 2120 1810 1810
q11 441 280 248 248
q12 697 429 298 298
q13 18232 3396 2793 2793
q14 264 262 237 237
q15 q16 813 772 701 701
q17 920 886 1021 886
q18 6905 5624 5516 5516
q19 1194 1440 1122 1122
q20 498 409 275 275
q21 6230 2743 2613 2613
q22 462 380 303 303
Total cold run time: 102399 ms
Total hot run time: 31205 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4623 4679 4591 4591
q2 q3 4865 5289 4606 4606
q4 2148 2317 1468 1468
q5 4825 4698 4662 4662
q6 233 173 127 127
q7 1866 1744 1380 1380
q8 2208 1925 1905 1905
q9 7222 7185 7156 7156
q10 4557 4422 3997 3997
q11 536 384 352 352
q12 716 721 512 512
q13 3012 3393 2779 2779
q14 266 276 251 251
q15 q16 676 703 625 625
q17 1264 1234 1229 1229
q18 7313 6907 6764 6764
q19 1145 1134 1105 1105
q20 2205 2211 1938 1938
q21 5329 4666 4504 4504
q22 518 454 420 420
Total cold run time: 55527 ms
Total hot run time: 50371 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant decimal precision [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4470493289
<details>
<summary>TPC-H: <b>Total hot run time: 31294 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 43b1a1432eb96745bdb80b377765e638551b4e56, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17684 3930 3838 3838
q2 q3 10778 1358 779 779
q4 4686 474 339 339
q5 7570 2248 2114 2114
q6 389 176 138 138
q7 938 783 619 619
q8 9482 1809 1632 1632
q9 7044 4914 4867 4867
q10 6442 2094 1812 1812
q11 441 276 250 250
q12 693 422 296 296
q13 18244 3462 2833 2833
q14 271 255 235 235
q15 q16 823 776 706 706
q17 977 938 927 927
q18 6849 5646 5541 5541
q19 1187 1270 1238 1238
q20 533 418 278 278
q21 6104 2833 2542 2542
q22 456 370 310 310
Total cold run time: 101591 ms
Total hot run time: 31294 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4695 4548 4689 4548
q2 q3 4887 5278 4672 4672
q4 2120 2217 1431 1431
q5 4873 4738 4655 4655
q6 235 183 129 129
q7 1888 1691 1578 1578
q8 2404 1935 1920 1920
q9 7305 7282 7213 7213
q10 4521 4430 3956 3956
q11 544 383 383 383
q12 709 721 507 507
q13 2993 3495 2787 2787
q14 270 287 247 247
q15 q16 677 709 624 624
q17 1272 1229 1251 1229
q18 7358 6840 6842 6840
q19 1152 1085 1127 1085
q20 2219 2219 1937 1937
q21 5365 4717 4535 4535
q22 529 472 411 411
Total cold run time: 56016 ms
Total hot run time: 50687 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant predefined decimal precision [doris]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4475505434
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant predefined decimal precision [doris]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon closed pull request #63322: [fix](be) Preserve Variant predefined decimal precision
URL: https://github.com/apache/doris/pull/63322
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant decimal precision [doris]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4471451190
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant decimal precision [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4470517569
<details>
<summary>TPC-DS: <b>Total hot run time: 171161 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 832ca14461bc0faf4fbf9b0d2a91c7df1111f393, data reload: false
query5 4338 652 527 527
query6 338 237 216 216
query7 4226 581 309 309
query8 335 239 225 225
query9 8815 4112 4075 4075
query10 451 354 296 296
query11 5778 2350 2218 2218
query12 182 132 125 125
query13 1276 623 441 441
query14 6019 5455 5122 5122
query14_1 4490 4438 4434 4434
query15 218 212 185 185
query16 983 480 460 460
query17 1160 755 636 636
query18 2502 507 377 377
query19 229 214 170 170
query20 137 150 131 131
query21 226 145 125 125
query22 13891 13611 13330 13330
query23 17171 16328 16064 16064
query23_1 16248 16273 16231 16231
query24 7515 1801 1316 1316
query24_1 1345 1325 1336 1325
query25 591 520 483 483
query26 1346 329 173 173
query27 2721 555 344 344
query28 4414 1969 1944 1944
query29 948 641 508 508
query30 303 235 198 198
query31 1113 1080 947 947
query32 91 78 74 74
query33 534 356 303 303
query34 1182 1159 663 663
query35 772 796 694 694
query36 1328 1324 1177 1177
query37 152 104 94 94
query38 3216 3172 3076 3076
query39 931 927 893 893
query39_1 875 876 866 866
query40 248 150 129 129
query41 67 63 64 63
query42 111 110 111 110
query43 331 346 291 291
query44
query45 209 202 196 196
query46 1053 1185 732 732
query47 2274 2272 2200 2200
query48 384 408 287 287
query49 634 502 386 386
query50 1034 358 246 246
query51 4352 4329 4204 4204
query52 108 112 97 97
query53 264 286 205 205
query54 310 268 255 255
query55 96 91 86 86
query56 310 311 300 300
query57 1398 1403 1272 1272
query58 300 278 280 278
query59 1607 1626 1454 1454
query60 329 328 320 320
query61 163 162 159 159
query62 667 646 547 547
query63 260 204 216 204
query64 2406 802 651 651
query65
query66 1707 494 368 368
query67 30012 30072 30026 30026
query68
query69 456 350 317 317
query70 1047 1070 1017 1017
query71 306 278 276 276
query72 3031 2929 2458 2458
query73 850 792 434 434
query74 5061 4985 4714 4714
query75 2691 2606 2264 2264
query76 2806 1161 777 777
query77 399 421 345 345
query78 12182 12305 11616 11616
query79 1438 994 740 740
query80 668 549 461 461
query81 457 277 243 243
query82 1416 160 120 120
query83 371 275 264 264
query84 267 146 112 112
query85 883 543 458 458
query86 389 327 311 311
query87 3462 3375 3227 3227
query88 3579 2685 2663 2663
query89 440 397 340 340
query90 1979 195 192 192
query91 183 179 144 144
query92 80 76 76 76
query93 1504 1429 942 942
query94 539 360 325 325
query95 662 483 359 359
query96 1042 781 326 326
query97 2706 2702 2576 2576
query98 244 234 234 234
query99 1136 1090 968 968
Total cold run time: 254164 ms
Total hot run time: 171161 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant decimal precision [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4470515240
<details>
<summary>TPC-H: <b>Total hot run time: 30897 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7885a6ce76862cca4cd5fd9cc18bd2d7185fb161, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17624 3801 3808 3801
q2 q3 10822 1327 851 851
q4 4680 469 357 357
q5 7595 2284 2081 2081
q6 240 177 137 137
q7 929 754 651 651
q8 9422 1780 1582 1582
q9 6585 4875 4841 4841
q10 6446 2111 1809 1809
q11 438 264 242 242
q12 699 429 291 291
q13 18190 3354 2742 2742
q14 268 254 234 234
q15 q16 814 781 702 702
q17 950 969 987 969
q18 6741 5630 5556 5556
q19 1181 1349 1052 1052
q20 529 410 258 258
q21 5902 2593 2437 2437
q22 432 368 304 304
Total cold run time: 100487 ms
Total hot run time: 30897 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4170 4076 4055 4055
q2 q3 4480 4864 4305 4305
q4 2090 2193 1375 1375
q5 4317 4301 4236 4236
q6 232 186 142 142
q7 2099 1928 1768 1768
q8 2488 2175 2103 2103
q9 8040 7818 7854 7818
q10 4638 4567 4118 4118
q11 600 417 385 385
q12 930 785 559 559
q13 3296 3552 2995 2995
q14 291 300 271 271
q15 q16 717 763 628 628
q17 1336 1298 1299 1298
q18 7931 7265 7050 7050
q19 1160 1067 1110 1067
q20 2216 2191 1930 1930
q21 5220 4542 4384 4384
q22 508 455 405 405
Total cold run time: 56759 ms
Total hot run time: 50892 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant decimal precision [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4472197391
# BE Regression && UT Coverage Report
Increment line coverage `78.53% (373/475)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63322_b2ec8138ab58c2542048d496e500a78733c50214_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63322_b2ec8138ab58c2542048d496e500a78733c50214_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 73.62% (27837/37810) |
| Line Coverage | 57.60% (301865/524098) |
| Region Coverage | 54.90% (252633/460156) |
| Branch Coverage | 56.38% (109135/193587) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant decimal precision [doris]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4471159728
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant decimal precision [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4471301313
<details>
<summary>TPC-DS: <b>Total hot run time: 168730 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 95ccafea2253112ebae2ad285d71df8be7689d9b, data reload: false
query5 4335 655 509 509
query6 325 212 198 198
query7 4229 522 299 299
query8 324 233 258 233
query9 8820 3970 3966 3966
query10 444 334 293 293
query11 5803 2417 2187 2187
query12 176 130 123 123
query13 1265 598 428 428
query14 5878 5330 4983 4983
query14_1 4290 4300 4314 4300
query15 208 203 178 178
query16 1056 425 411 411
query17 975 737 599 599
query18 2481 483 362 362
query19 220 230 161 161
query20 136 133 134 133
query21 212 136 120 120
query22 13578 13442 13366 13366
query23 17163 16228 15996 15996
query23_1 16072 16100 16157 16100
query24 7496 1709 1315 1315
query24_1 1305 1304 1314 1304
query25 548 477 416 416
query26 1306 313 168 168
query27 2729 542 326 326
query28 4481 1921 1921 1921
query29 975 608 489 489
query30 300 238 198 198
query31 1110 1057 934 934
query32 88 76 71 71
query33 554 343 288 288
query34 1171 1129 626 626
query35 743 788 671 671
query36 1336 1345 1162 1162
query37 152 104 90 90
query38 3178 3125 3036 3036
query39 933 926 892 892
query39_1 873 889 875 875
query40 237 148 128 128
query41 72 69 69 69
query42 114 113 112 112
query43 319 323 286 286
query44
query45 211 204 204 204
query46 1100 1193 732 732
query47 2348 2351 2192 2192
query48 400 426 294 294
query49 669 521 405 405
query50 997 358 253 253
query51 4316 4223 4217 4217
query52 106 109 96 96
query53 262 284 211 211
query54 336 302 289 289
query55 96 92 88 88
query56 308 333 315 315
query57 1417 1416 1326 1326
query58 302 288 282 282
query59 1535 1655 1449 1449
query60 330 325 327 325
query61 187 179 185 179
query62 671 635 572 572
query63 248 202 212 202
query64 2476 866 692 692
query65
query66 1763 502 366 366
query67 29409 30019 29838 29838
query68
query69 457 349 310 310
query70 1058 964 976 964
query71 304 280 307 280
query72 3006 2657 2419 2419
query73 818 749 418 418
query74 5063 4880 4731 4731
query75 2672 2601 2259 2259
query76 2284 1153 762 762
query77 395 408 340 340
query78 12213 12153 11689 11689
query79 1500 1014 753 753
query80 645 541 456 456
query81 447 282 244 244
query82 1353 158 119 119
query83 350 285 253 253
query84 307 140 106 106
query85 886 542 465 465
query86 388 336 307 307
query87 3402 3321 3222 3222
query88 3542 2682 2641 2641
query89 452 389 351 351
query90 1958 174 181 174
query91 182 172 146 146
query92 77 81 78 78
query93 1549 1476 831 831
query94 542 330 297 297
query95 717 474 347 347
query96 1075 767 361 361
query97 2679 2670 2569 2569
query98 232 226 227 226
query99 1132 1084 998 998
Total cold run time: 251895 ms
Total hot run time: 168730 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant decimal precision [doris]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4471288351
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant decimal precision [doris]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4471288352
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant decimal precision [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4471332709
<details>
<summary>TPC-H: <b>Total hot run time: 31342 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1e5d7d35ae5f95302d9c310ad534a8d0cafd09d5, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17595 3915 4019 3915
q2 q3 10759 1411 807 807
q4 4680 472 343 343
q5 7544 2344 2126 2126
q6 241 196 147 147
q7 950 798 642 642
q8 9669 1672 1574 1574
q9 5275 5011 5098 5011
q10 6390 2077 1795 1795
q11 480 274 249 249
q12 643 434 295 295
q13 18107 3412 2722 2722
q14 270 254 238 238
q15 q16 822 765 701 701
q17 953 960 854 854
q18 6955 5649 5555 5555
q19 1299 1260 1030 1030
q20 541 568 316 316
q21 6016 2830 2694 2694
q22 480 384 328 328
Total cold run time: 99669 ms
Total hot run time: 31342 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4954 4621 4529 4529
q2 q3 4863 5262 4579 4579
q4 2117 2232 1398 1398
q5 4737 4810 4690 4690
q6 236 175 131 131
q7 1915 1820 1518 1518
q8 2353 2139 2183 2139
q9 7873 7617 7318 7318
q10 4489 4400 3983 3983
q11 534 414 359 359
q12 711 730 505 505
q13 3030 3436 2827 2827
q14 277 278 253 253
q15 q16 675 694 622 622
q17 1267 1237 1262 1237
q18 7346 6938 6742 6742
q19 1135 1097 1127 1097
q20 2203 2218 1949 1949
q21 5392 4690 4661 4661
q22 522 460 416 416
Total cold run time: 56629 ms
Total hot run time: 50953 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant decimal precision [doris]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4471350382
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant decimal precision [doris]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4470383598
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant decimal precision [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4471373313
<details>
<summary>TPC-DS: <b>Total hot run time: 169675 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1e5d7d35ae5f95302d9c310ad534a8d0cafd09d5, data reload: false
query5 4320 667 531 531
query6 340 227 203 203
query7 4223 567 306 306
query8 328 234 227 227
query9 8845 4055 4018 4018
query10 452 341 303 303
query11 5754 2423 2211 2211
query12 178 128 128 128
query13 1266 607 428 428
query14 6031 5395 5028 5028
query14_1 4330 4380 4353 4353
query15 213 200 182 182
query16 1019 439 442 439
query17 942 721 575 575
query18 2448 485 358 358
query19 211 203 161 161
query20 134 142 126 126
query21 216 137 120 120
query22 13560 13639 13387 13387
query23 17242 16446 15978 15978
query23_1 16203 16173 16160 16160
query24 7407 1733 1292 1292
query24_1 1302 1320 1306 1306
query25 566 484 434 434
query26 1316 335 192 192
query27 2684 573 337 337
query28 4569 1953 1907 1907
query29 993 630 526 526
query30 309 243 207 207
query31 1114 1067 943 943
query32 105 79 79 79
query33 548 363 335 335
query34 1180 1148 648 648
query35 768 773 670 670
query36 1303 1349 1092 1092
query37 151 107 92 92
query38 3242 3138 3078 3078
query39 936 916 901 901
query39_1 897 877 879 877
query40 229 165 135 135
query41 83 74 63 63
query42 111 112 109 109
query43 331 329 303 303
query44
query45 208 196 192 192
query46 1048 1168 734 734
query47 2330 2263 2125 2125
query48 405 432 285 285
query49 628 496 377 377
query50 963 345 274 274
query51 4396 4239 4231 4231
query52 116 113 96 96
query53 256 285 207 207
query54 324 263 263 263
query55 95 91 85 85
query56 301 306 310 306
query57 1389 1374 1281 1281
query58 335 271 276 271
query59 1556 1598 1442 1442
query60 312 327 319 319
query61 195 162 153 153
query62 664 639 545 545
query63 239 204 210 204
query64 2445 854 652 652
query65
query66 1745 463 357 357
query67 30068 30008 29412 29412
query68
query69 469 333 300 300
query70 952 1013 1013 1013
query71 311 267 271 267
query72 3168 2765 2418 2418
query73 852 737 423 423
query74 5079 4897 4758 4758
query75 2698 2607 2256 2256
query76 2259 1164 794 794
query77 418 421 326 326
query78 12247 11912 11595 11595
query79 1261 1043 739 739
query80 596 550 461 461
query81 456 279 245 245
query82 242 161 120 120
query83 274 281 248 248
query84 288 134 115 115
query85 890 527 441 441
query86 368 346 357 346
query87 3368 3386 3243 3243
query88 3532 2680 2692 2680
query89 430 388 337 337
query90 2156 187 183 183
query91 178 169 138 138
query92 78 76 73 73
query93 1568 1439 887 887
query94 548 352 281 281
query95 689 486 342 342
query96 1028 797 341 341
query97 2679 2727 2580 2580
query98 236 227 223 223
query99 1093 1104 990 990
Total cold run time: 251776 ms
Total hot run time: 169675 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant decimal precision [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4470761620
<details>
<summary>TPC-DS: <b>Total hot run time: 169308 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 85fcf569e2cba61ca79f53700ba241f8851c2bd3, data reload: false
query5 4330 653 516 516
query6 326 233 201 201
query7 4239 617 316 316
query8 330 236 225 225
query9 8820 4007 4015 4007
query10 444 330 286 286
query11 5857 2424 2222 2222
query12 189 127 130 127
query13 1326 592 431 431
query14 6182 5334 5019 5019
query14_1 4304 4351 4309 4309
query15 211 203 182 182
query16 1010 464 441 441
query17 1116 709 602 602
query18 2704 496 348 348
query19 215 200 166 166
query20 143 130 127 127
query21 218 141 114 114
query22 13535 13517 13550 13517
query23 17296 16446 16140 16140
query23_1 16130 16321 16148 16148
query24 7470 1774 1292 1292
query24_1 1300 1303 1308 1303
query25 579 509 440 440
query26 1311 318 176 176
query27 2678 580 341 341
query28 4442 1961 1957 1957
query29 1039 630 527 527
query30 306 247 205 205
query31 1132 1062 944 944
query32 98 82 82 82
query33 541 366 306 306
query34 1197 1122 654 654
query35 772 780 683 683
query36 1346 1319 1173 1173
query37 164 119 97 97
query38 3249 3132 3059 3059
query39 933 915 902 902
query39_1 884 896 886 886
query40 231 149 127 127
query41 72 73 72 72
query42 111 112 113 112
query43 327 329 284 284
query44
query45 224 204 197 197
query46 1129 1234 740 740
query47 2372 2291 2207 2207
query48 397 445 297 297
query49 647 504 416 416
query50 1043 360 251 251
query51 4347 4261 4238 4238
query52 110 110 96 96
query53 263 293 210 210
query54 322 284 278 278
query55 99 92 91 91
query56 325 326 320 320
query57 1416 1403 1313 1313
query58 306 278 278 278
query59 1574 1633 1479 1479
query60 326 344 320 320
query61 191 212 149 149
query62 681 626 564 564
query63 243 200 207 200
query64 2346 792 636 636
query65
query66 1664 481 359 359
query67 30041 29861 29881 29861
query68
query69 455 338 304 304
query70 1055 1002 947 947
query71 312 278 268 268
query72 2994 2685 2448 2448
query73 835 784 447 447
query74 5075 4890 4754 4754
query75 2700 2597 2236 2236
query76 2318 1157 763 763
query77 409 414 337 337
query78 12172 12104 11573 11573
query79 1437 1024 764 764
query80 1302 543 448 448
query81 528 276 245 245
query82 1048 155 120 120
query83 354 274 252 252
query84 312 143 114 114
query85 937 535 488 488
query86 455 343 343 343
query87 3502 3355 3178 3178
query88 3574 2691 2647 2647
query89 451 384 338 338
query90 1953 184 180 180
query91 182 167 137 137
query92 78 80 80 80
query93 1556 1496 830 830
query94 727 362 319 319
query95 688 482 347 347
query96 1022 793 373 373
query97 2734 2674 2553 2553
query98 232 232 233 232
query99 1127 1082 982 982
Total cold run time: 254607 ms
Total hot run time: 169308 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant decimal precision [doris]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4470763984
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant decimal precision [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #63322:
URL: https://github.com/apache/doris/pull/63322#discussion_r3254488698
##########
be/src/exec/common/variant_util.cpp:
##########
@@ -2247,6 +2285,7 @@ Status parse_and_materialize_variant_columns(Block& block, const TabletSchema& t
}
// if doc mode is not enabled, no need to parse to doc value column
if (!column.variant_enable_doc_mode()) {
+ configure_decimal_number_preserve_paths(column, &configs[i]);
configs[i].parse_to = ParseConfig::ParseTo::OnlySubcolumns;
Review Comment:
This only configures raw-number preservation for non-doc-mode Variant columns. Doc-mode Variant columns can still have predefined typed paths (for example the existing `variant_p0/doc_mode/predefine` suites use `default_variant_enable_doc_mode = true` with `MATCH_NAME` paths), and that insert path sets `ParseTo::OnlyDocValueColumn` below. In that mode `parse_json_to_variant()` still serializes each field into the doc-value binary column from `getValueAsField()` without these decimal paths configured, so a high-precision `decimalv3` typed path is stored from the rounded `TYPE_DOUBLE` and later materialization/cast cannot recover the original token. Please configure the decimal preserve paths before the doc-mode split as well, and add a doc-mode predefined decimal regression case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant decimal precision [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4470516966
<details>
<summary>TPC-DS: <b>Total hot run time: 168854 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 43b1a1432eb96745bdb80b377765e638551b4e56, data reload: false
query5 4341 657 522 522
query6 321 211 201 201
query7 4207 549 301 301
query8 332 247 220 220
query9 8841 4007 3999 3999
query10 451 352 292 292
query11 5792 2419 2222 2222
query12 188 128 124 124
query13 1264 585 417 417
query14 5939 5330 5015 5015
query14_1 4328 4286 4331 4286
query15 229 201 184 184
query16 985 457 434 434
query17 933 710 621 621
query18 2451 494 375 375
query19 223 225 174 174
query20 139 137 128 128
query21 220 143 119 119
query22 13655 13480 13397 13397
query23 17232 16455 15923 15923
query23_1 16054 16198 16067 16067
query24 7627 1831 1308 1308
query24_1 1316 1287 1307 1287
query25 557 501 439 439
query26 1314 317 174 174
query27 2680 578 335 335
query28 4478 1969 1942 1942
query29 1031 649 522 522
query30 298 250 206 206
query31 1111 1060 930 930
query32 107 80 73 73
query33 536 374 300 300
query34 1154 1136 635 635
query35 769 789 694 694
query36 1333 1343 1104 1104
query37 157 105 93 93
query38 3230 3121 3032 3032
query39 922 936 904 904
query39_1 883 865 883 865
query40 237 154 130 130
query41 76 73 70 70
query42 115 112 117 112
query43 327 329 304 304
query44
query45 214 195 190 190
query46 1050 1209 728 728
query47 2335 2337 2196 2196
query48 403 445 303 303
query49 637 492 388 388
query50 1003 342 257 257
query51 4392 4299 4289 4289
query52 104 107 101 101
query53 257 289 203 203
query54 323 272 265 265
query55 100 90 92 90
query56 311 312 293 293
query57 1414 1385 1289 1289
query58 300 263 269 263
query59 1528 1606 1435 1435
query60 321 326 314 314
query61 156 154 150 150
query62 670 625 539 539
query63 242 203 208 203
query64 2415 820 654 654
query65
query66 1767 469 358 358
query67 29934 29866 29840 29840
query68
query69 466 340 315 315
query70 1034 1015 932 932
query71 304 275 282 275
query72 2976 2691 2373 2373
query73 854 786 428 428
query74 5079 4887 4738 4738
query75 2670 2581 2259 2259
query76 2302 1121 775 775
query77 392 404 334 334
query78 12170 11989 11617 11617
query79 1481 1056 738 738
query80 772 561 457 457
query81 469 274 238 238
query82 1362 156 119 119
query83 350 274 254 254
query84 306 142 110 110
query85 928 534 451 451
query86 420 336 329 329
query87 3380 3397 3203 3203
query88 3526 2662 2622 2622
query89 447 389 337 337
query90 1899 181 180 180
query91 179 165 139 139
query92 82 80 74 74
query93 1488 1489 865 865
query94 598 338 312 312
query95 691 466 339 339
query96 987 780 329 329
query97 2700 2701 2561 2561
query98 240 234 230 230
query99 1126 1089 983 983
Total cold run time: 252772 ms
Total hot run time: 168854 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant decimal precision [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4471763320
# BE UT Coverage Report
Increment line coverage `71.58% (330/461)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/45d9060c080f192ecb190462a4b01b6485daad2a_45d9060c080f192ecb190462a4b01b6485daad2a/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/45d9060c080f192ecb190462a4b01b6485daad2a_45d9060c080f192ecb190462a4b01b6485daad2a/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 53.51% (20661/38609) |
| Line Coverage | 37.17% (195329/525461) |
| Region Coverage | 33.56% (152929/455729) |
| Branch Coverage | 34.60% (66729/192847) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant predefined decimal precision [doris]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4474628414
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant predefined decimal precision [doris]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4475936922
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant predefined decimal precision [doris]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4475103411
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant predefined decimal precision [doris]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4475505367
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant decimal precision [doris]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #63322:
URL: https://github.com/apache/doris/pull/63322#discussion_r3254419851
##########
be/src/util/json/simd_json_parser.h:
##########
@@ -20,71 +20,82 @@
#pragma once
-#include <rapidjson/document.h>
#include <simdjson.h>
+#include <memory>
+#include <string>
+#include <string_view>
+#include <type_traits>
+#include <utility>
+#include <vector>
+
Review Comment:
simd_json_parser.h uses assert() in Array::operator[] and Object::operator[] but the header does not include <cassert>, so compilation can fail depending on include order. Add #include <cassert> in this header to make it self-contained.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant decimal precision [doris]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4470266529
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant predefined decimal precision [doris]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4475845011
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant decimal precision [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4470516561
<details>
<summary>TPC-DS: <b>Total hot run time: 170350 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9ccf680bcc7622c4472bb012526de8e07acd5055, data reload: false
query5 4295 634 514 514
query6 317 215 192 192
query7 4215 539 328 328
query8 320 226 217 217
query9 8826 4049 4044 4044
query10 450 336 311 311
query11 5804 2355 2171 2171
query12 179 125 127 125
query13 1264 651 414 414
query14 5918 5308 5013 5013
query14_1 4345 4306 4303 4303
query15 207 197 181 181
query16 1030 486 463 463
query17 1107 703 603 603
query18 2727 479 344 344
query19 215 200 156 156
query20 138 139 128 128
query21 214 137 118 118
query22 13577 13590 13332 13332
query23 17161 16496 16137 16137
query23_1 16285 16302 16230 16230
query24 7813 1766 1288 1288
query24_1 1310 1285 1290 1285
query25 561 488 414 414
query26 1307 309 173 173
query27 2706 554 347 347
query28 4435 1938 1942 1938
query29 969 626 508 508
query30 305 242 195 195
query31 1107 1073 934 934
query32 87 76 72 72
query33 541 373 297 297
query34 1165 1157 661 661
query35 769 787 686 686
query36 1346 1361 1183 1183
query37 145 111 92 92
query38 3210 3126 3054 3054
query39 935 943 906 906
query39_1 887 875 876 875
query40 230 144 125 125
query41 66 65 61 61
query42 109 111 107 107
query43 325 325 286 286
query44
query45 213 202 196 196
query46 1069 1166 724 724
query47 2385 2339 2216 2216
query48 398 420 300 300
query49 655 521 411 411
query50 980 352 257 257
query51 4345 4315 4234 4234
query52 108 108 99 99
query53 260 288 222 222
query54 335 284 274 274
query55 98 91 86 86
query56 308 316 317 316
query57 1435 1429 1321 1321
query58 307 284 283 283
query59 1566 1662 1455 1455
query60 323 336 322 322
query61 182 178 175 175
query62 669 650 556 556
query63 257 211 214 211
query64 2424 843 670 670
query65
query66 1693 501 373 373
query67 29911 29904 29795 29795
query68
query69 470 372 321 321
query70 1064 994 1000 994
query71 308 274 273 273
query72 3225 2684 2406 2406
query73 857 725 396 396
query74 5053 4887 4718 4718
query75 2671 2586 2226 2226
query76 2284 1144 733 733
query77 397 397 337 337
query78 12050 12202 11578 11578
query79 1472 985 746 746
query80 1331 541 505 505
query81 536 279 237 237
query82 1091 160 123 123
query83 344 272 248 248
query84 260 136 109 109
query85 932 542 449 449
query86 447 373 302 302
query87 3506 3349 3205 3205
query88 3506 2653 2603 2603
query89 448 390 338 338
query90 1875 171 174 171
query91 178 168 137 137
query92 78 76 71 71
query93 1577 1471 820 820
query94 708 350 304 304
query95 700 402 347 347
query96 1077 790 353 353
query97 2784 2747 2612 2612
query98 238 239 227 227
query99 1107 1144 1025 1025
Total cold run time: 254353 ms
Total hot run time: 170350 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant decimal precision [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4470492567
<details>
<summary>TPC-H: <b>Total hot run time: 31288 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9ccf680bcc7622c4472bb012526de8e07acd5055, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17710 3972 3884 3884
q2 q3 10805 1389 829 829
q4 4686 479 344 344
q5 7561 2298 2145 2145
q6 362 178 139 139
q7 971 774 652 652
q8 9641 1721 1729 1721
q9 6702 4974 4944 4944
q10 6470 2150 1828 1828
q11 454 273 247 247
q12 700 420 295 295
q13 18248 3545 2804 2804
q14 264 255 237 237
q15 q16 823 776 705 705
q17 934 877 947 877
q18 6949 5687 5721 5687
q19 1250 1270 1064 1064
q20 496 418 255 255
q21 5749 2580 2326 2326
q22 444 377 305 305
Total cold run time: 101219 ms
Total hot run time: 31288 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4201 4114 4127 4114
q2 q3 4514 4886 4366 4366
q4 2138 2196 1421 1421
q5 4426 4305 4617 4305
q6 249 193 146 146
q7 2061 1850 1599 1599
q8 2493 2099 2144 2099
q9 7794 7881 7766 7766
q10 4558 4497 4326 4326
q11 617 456 385 385
q12 758 732 535 535
q13 3258 3683 3044 3044
q14 292 318 294 294
q15 q16 720 768 658 658
q17 1308 1322 1317 1317
q18 7812 7413 6904 6904
q19 1127 1086 1102 1086
q20 2233 2233 1938 1938
q21 5373 4660 4564 4564
q22 529 474 406 406
Total cold run time: 56461 ms
Total hot run time: 51273 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant decimal precision [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4470913351
<details>
<summary>TPC-H: <b>Total hot run time: 31478 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c34c703b31a24812923fba9197fe4db35efe1ccb, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17710 4078 4033 4033
q2 q3 10771 1374 799 799
q4 4683 467 343 343
q5 7667 2220 2093 2093
q6 323 188 142 142
q7 896 777 631 631
q8 9424 1695 1678 1678
q9 6930 4909 4918 4909
q10 6522 2212 1842 1842
q11 447 290 263 263
q12 690 483 301 301
q13 18202 3443 2739 2739
q14 268 259 247 247
q15 q16 820 764 714 714
q17 976 935 912 912
q18 6864 5733 5474 5474
q19 1167 1472 1205 1205
q20 542 429 290 290
q21 6170 2812 2551 2551
q22 455 395 312 312
Total cold run time: 101527 ms
Total hot run time: 31478 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4942 4849 5060 4849
q2 q3 4808 5226 4646 4646
q4 2265 2249 1376 1376
q5 4949 4815 4699 4699
q6 235 182 134 134
q7 1852 1729 1605 1605
q8 2575 2074 2006 2006
q9 7307 7317 7271 7271
q10 4450 4392 3964 3964
q11 560 419 382 382
q12 703 716 504 504
q13 3021 3446 2805 2805
q14 283 270 251 251
q15 q16 688 706 606 606
q17 1266 1242 1244 1242
q18 7617 6753 7028 6753
q19 1117 1083 1121 1083
q20 2224 2215 1938 1938
q21 5430 4753 4577 4577
q22 545 473 412 412
Total cold run time: 56837 ms
Total hot run time: 51103 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant decimal precision [doris]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4470881546
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant predefined decimal precision [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #63322:
URL: https://github.com/apache/doris/pull/63322#discussion_r3256322828
##########
be/src/util/json/json_parser.h:
##########
@@ -53,16 +55,31 @@ Field getValueAsField(const Element& element) {
if (element.isUInt64()) {
return Field::create_field<TYPE_LARGEINT>(static_cast<int128_t>(element.getUInt64()));
}
+ if (element.isInt128()) {
+ if (preserve_number_as_string) {
+ return Field::create_field<TYPE_STRING>(String(element.getRawNumber()));
+ }
+ return Field::create_field<TYPE_LARGEINT>(element.getInt128());
+ }
+ if (element.isBigInteger()) {
+ if (preserve_number_as_string) {
+ return Field::create_field<TYPE_STRING>(String(element.getRawNumber()));
+ }
+ return {};
+ }
Review Comment:
When raw preservation is enabled for any decimal typed path in the Variant column, the whole document is parsed through the ondemand `Node` tree. For an unrelated path containing a JSON integer larger than UInt64, `isBigInteger()` reaches this branch with `preserve_number_as_string == false` and returns a null `Field`, so a value that used to be present in an untyped path is silently stored as NULL/missing just because another path in the schema needs decimal preservation. For example, with a decimal typed path `a`, loading `{"a": 1.23, "b": 340282366920938463463374607431768211456}` loses `b`. Please preserve non-decimal big integers as a string (or otherwise keep the previous behavior) instead of returning null, and add a regression where a decimal typed path coexists with an unrelated oversized integer path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant decimal precision [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4470179388
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant decimal precision [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4470550270
<details>
<summary>TPC-DS: <b>Total hot run time: 168668 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7885a6ce76862cca4cd5fd9cc18bd2d7185fb161, data reload: false
query5 4310 656 525 525
query6 333 224 209 209
query7 4240 590 300 300
query8 322 258 213 213
query9 8805 3959 3978 3959
query10 447 327 294 294
query11 5769 2359 2214 2214
query12 187 130 133 130
query13 1266 631 431 431
query14 5938 5333 5005 5005
query14_1 4320 4306 4337 4306
query15 219 202 177 177
query16 992 444 413 413
query17 1009 728 584 584
query18 2448 482 351 351
query19 207 201 174 174
query20 133 135 125 125
query21 209 137 118 118
query22 13545 13656 13407 13407
query23 17150 16323 15973 15973
query23_1 16131 16195 16102 16102
query24 7518 1781 1315 1315
query24_1 1320 1260 1307 1260
query25 541 472 415 415
query26 1313 317 169 169
query27 2746 540 339 339
query28 4500 1946 1928 1928
query29 994 613 507 507
query30 313 247 197 197
query31 1115 1057 954 954
query32 85 74 70 70
query33 533 346 280 280
query34 1155 1123 627 627
query35 773 783 676 676
query36 1337 1311 1182 1182
query37 162 105 92 92
query38 3238 3122 3056 3056
query39 929 923 909 909
query39_1 886 884 880 880
query40 236 151 131 131
query41 74 70 70 70
query42 112 115 114 114
query43 326 327 288 288
query44
query45 218 203 193 193
query46 1075 1188 742 742
query47 2371 2392 2145 2145
query48 402 394 305 305
query49 645 499 394 394
query50 985 344 245 245
query51 4327 4241 4176 4176
query52 105 109 97 97
query53 265 291 209 209
query54 333 281 280 280
query55 96 91 86 86
query56 317 321 318 318
query57 1397 1409 1313 1313
query58 306 289 285 285
query59 1545 1680 1416 1416
query60 333 337 321 321
query61 184 180 180 180
query62 680 644 556 556
query63 245 210 213 210
query64 2491 872 705 705
query65
query66 1724 505 372 372
query67 30020 29840 29695 29695
query68
query69 471 348 309 309
query70 1033 1011 938 938
query71 307 283 274 274
query72 3114 2618 2415 2415
query73 826 763 400 400
query74 5067 4918 4717 4717
query75 2658 2573 2266 2266
query76 2287 1141 816 816
query77 404 406 338 338
query78 12120 12067 11718 11718
query79 1467 1068 723 723
query80 1312 540 467 467
query81 527 283 238 238
query82 980 196 126 126
query83 343 274 247 247
query84 310 136 113 113
query85 919 530 461 461
query86 464 343 332 332
query87 3398 3402 3202 3202
query88 3554 2668 2640 2640
query89 447 386 340 340
query90 1925 183 182 182
query91 179 167 148 148
query92 79 79 75 75
query93 1567 1406 869 869
query94 710 346 304 304
query95 694 377 340 340
query96 1012 753 327 327
query97 2666 2725 2550 2550
query98 249 226 245 226
query99 1103 1102 980 980
Total cold run time: 253475 ms
Total hot run time: 168668 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant decimal precision [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4471274644
<details>
<summary>TPC-H: <b>Total hot run time: 30935 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 95ccafea2253112ebae2ad285d71df8be7689d9b, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17920 3941 3909 3909
q2 q3 10782 1407 802 802
q4 4687 471 356 356
q5 7587 2266 2101 2101
q6 232 174 138 138
q7 936 788 632 632
q8 9386 1753 1569 1569
q9 5075 4879 4869 4869
q10 6401 2134 1817 1817
q11 431 270 240 240
q12 624 440 302 302
q13 18084 3369 2686 2686
q14 260 254 235 235
q15 q16 825 773 710 710
q17 991 865 909 865
q18 6860 5716 5726 5716
q19 1361 1212 1097 1097
q20 533 410 266 266
q21 6103 2624 2324 2324
q22 418 360 301 301
Total cold run time: 99496 ms
Total hot run time: 30935 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4158 4115 4082 4082
q2 q3 4467 4892 4283 4283
q4 2117 2189 1385 1385
q5 4372 4261 4273 4261
q6 225 177 130 130
q7 1733 2140 1821 1821
q8 2590 2148 2122 2122
q9 7911 8092 7663 7663
q10 4548 4501 4042 4042
q11 552 408 375 375
q12 728 720 525 525
q13 3298 3687 2951 2951
q14 315 310 274 274
q15 q16 718 733 636 636
q17 1330 1296 1290 1290
q18 8015 7320 7179 7179
q19 1163 1131 1159 1131
q20 2200 2215 1932 1932
q21 5288 4573 4439 4439
q22 516 464 415 415
Total cold run time: 56244 ms
Total hot run time: 50936 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant decimal precision [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4471644158
<details>
<summary>TPC-DS: <b>Total hot run time: 169664 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b2ec8138ab58c2542048d496e500a78733c50214, data reload: false
query5 4313 653 512 512
query6 335 218 193 193
query7 4309 568 305 305
query8 324 233 226 226
query9 8814 3969 4001 3969
query10 443 345 293 293
query11 5785 2438 2223 2223
query12 179 129 123 123
query13 1271 636 446 446
query14 5982 5352 5043 5043
query14_1 4398 4297 4331 4297
query15 213 207 181 181
query16 987 433 426 426
query17 938 707 585 585
query18 2446 507 347 347
query19 208 201 152 152
query20 136 132 130 130
query21 212 136 119 119
query22 13723 13770 13339 13339
query23 17210 16383 16012 16012
query23_1 16176 16112 16232 16112
query24 7414 1755 1308 1308
query24_1 1292 1299 1311 1299
query25 579 483 413 413
query26 1313 311 170 170
query27 2708 563 337 337
query28 4525 1946 1939 1939
query29 1000 615 494 494
query30 305 239 199 199
query31 1124 1060 939 939
query32 86 73 69 69
query33 543 349 296 296
query34 1156 1115 638 638
query35 775 786 678 678
query36 1327 1308 1162 1162
query37 150 103 91 91
query38 3207 3132 3069 3069
query39 933 926 888 888
query39_1 898 886 867 867
query40 224 146 123 123
query41 66 62 66 62
query42 112 110 111 110
query43 319 326 286 286
query44
query45 208 205 192 192
query46 1090 1177 720 720
query47 2333 2378 2235 2235
query48 413 422 284 284
query49 644 482 386 386
query50 911 343 255 255
query51 4317 4295 4276 4276
query52 104 103 94 94
query53 253 279 209 209
query54 306 266 264 264
query55 93 90 91 90
query56 302 299 299 299
query57 1455 1402 1324 1324
query58 295 268 270 268
query59 1584 1605 1440 1440
query60 326 324 331 324
query61 180 181 183 181
query62 712 619 564 564
query63 244 199 204 199
query64 2441 799 671 671
query65
query66 1711 483 354 354
query67 30082 29352 29297 29297
query68
query69 450 340 296 296
query70 1064 955 979 955
query71 310 270 259 259
query72 3027 2703 2505 2505
query73 847 760 439 439
query74 5104 4967 4778 4778
query75 2695 2621 2297 2297
query76 2279 1175 791 791
query77 394 419 341 341
query78 12190 12097 11548 11548
query79 1223 1013 748 748
query80 616 589 495 495
query81 457 278 246 246
query82 239 166 123 123
query83 281 282 254 254
query84 292 143 113 113
query85 948 629 532 532
query86 359 365 324 324
query87 3379 3363 3218 3218
query88 3507 2685 2667 2667
query89 413 393 339 339
query90 2202 178 179 178
query91 180 168 141 141
query92 77 78 73 73
query93 1439 1449 845 845
query94 541 348 321 321
query95 656 471 353 353
query96 1069 809 329 329
query97 2713 2688 2609 2609
query98 241 236 230 230
query99 1129 1087 943 943
Total cold run time: 251826 ms
Total hot run time: 169664 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant predefined decimal precision [doris]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4474525628
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant decimal precision [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #63322:
URL: https://github.com/apache/doris/pull/63322#discussion_r3254523071
##########
be/src/exprs/function/cast/cast_to_decimal.h:
##########
@@ -703,10 +718,16 @@ class CastToImpl<Mode, DataTypeString, ToDataType> : public CastToBase {
} else if constexpr (Mode == CastModeType::StrictMode) {
// WON'T write nulls to nullable_col_to, just raise errors. null_map is only used to skip invalid rows
RETURN_IF_ERROR(serde->from_string_strict_mode_batch(
- *col_from, nullable_col_to->get_nested_column(), {}, null_map));
Review Comment:
This starts feeding a nullable source null map into `DataTypeDecimalSerDe::from_string_strict_mode_batch()`, but that serde advances `current_offset` only after parsing a row and simply `continue`s when `null_map[i]` is true. Null positions in `ColumnNullable` are allowed to carry arbitrary nested string payloads, so after a skipped null with non-empty payload the next non-null row is parsed from the wrong byte range. This can corrupt/fail typed Variant decimal materialization when a nullable string subcolumn is cast to decimal. Please either fix the decimal serde to advance offsets for skipped rows or avoid this strict batch path with nullable string payloads, and add a nullable/missing typed decimal case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant decimal precision [doris]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4470808028
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant decimal precision [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4471179889
<details>
<summary>TPC-H: <b>Total hot run time: 31352 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 149c7f4f34ee4e10b9562d7dfea1bc11ad5227ea, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17759 3966 3900 3900
q2 q3 10766 1343 837 837
q4 4679 465 345 345
q5 7536 2240 2097 2097
q6 267 178 141 141
q7 936 797 614 614
q8 9352 1727 1620 1620
q9 6522 4905 4924 4905
q10 6459 2117 1784 1784
q11 438 270 246 246
q12 696 429 301 301
q13 18214 3303 2815 2815
q14 266 257 235 235
q15 q16 815 763 707 707
q17 933 907 980 907
q18 6937 5804 5725 5725
q19 1200 1226 1111 1111
q20 509 411 256 256
q21 5656 2640 2497 2497
q22 425 368 309 309
Total cold run time: 100365 ms
Total hot run time: 31352 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4309 4285 4233 4233
q2 q3 4487 4952 4278 4278
q4 2098 2203 1380 1380
q5 4425 4280 4307 4280
q6 225 179 137 137
q7 2316 1925 1657 1657
q8 2481 2174 2086 2086
q9 7957 7838 7717 7717
q10 4511 4525 4094 4094
q11 596 586 381 381
q12 705 734 524 524
q13 3436 3609 3060 3060
q14 291 295 266 266
q15 q16 752 737 668 668
q17 1370 1334 1343 1334
q18 8021 7486 6862 6862
q19 1105 1078 1095 1078
q20 2224 2217 1940 1940
q21 5362 4696 4576 4576
q22 541 485 423 423
Total cold run time: 57212 ms
Total hot run time: 50974 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant predefined decimal precision [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4475057793
# BE UT Coverage Report
Increment line coverage `78.95% (285/361)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/67a3b9fc0f89c62477627b265b9cc140f1c12659_67a3b9fc0f89c62477627b265b9cc140f1c12659/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/67a3b9fc0f89c62477627b265b9cc140f1c12659_67a3b9fc0f89c62477627b265b9cc140f1c12659/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 53.51% (20659/38606) |
| Line Coverage | 37.18% (195356/525397) |
| Region Coverage | 33.55% (152903/455767) |
| Branch Coverage | 34.60% (66720/192818) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant predefined decimal precision [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4477109938
<details>
<summary>TPC-H: <b>Total hot run time: 31157 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1f40adb4feed235b8c3c5c6e215682a37a4efb68, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17640 3885 3854 3854
q2 q3 10825 1329 777 777
q4 4689 476 344 344
q5 7582 2291 2064 2064
q6 241 177 136 136
q7 937 763 636 636
q8 9346 1899 1551 1551
q9 5077 4886 4985 4886
q10 6375 2077 1812 1812
q11 441 271 249 249
q12 629 429 295 295
q13 18073 3391 2773 2773
q14 262 264 231 231
q15 q16 813 787 706 706
q17 989 893 1031 893
q18 6918 5786 5449 5449
q19 1318 1279 1126 1126
q20 533 398 291 291
q21 6554 2792 2765 2765
q22 468 383 319 319
Total cold run time: 99710 ms
Total hot run time: 31157 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4748 4562 4445 4445
q2 q3 4879 5345 4607 4607
q4 2114 2185 1429 1429
q5 4732 4843 4631 4631
q6 236 182 126 126
q7 1856 1770 1579 1579
q8 2405 2033 2008 2008
q9 7789 7608 7134 7134
q10 4469 4416 4032 4032
q11 520 377 347 347
q12 718 719 520 520
q13 3022 3398 2794 2794
q14 271 272 243 243
q15 q16 683 694 609 609
q17 1258 1228 1212 1212
q18 7226 6776 6830 6776
q19 1093 1109 1111 1109
q20 2237 2194 1924 1924
q21 5274 4603 4496 4496
q22 542 477 422 422
Total cold run time: 56072 ms
Total hot run time: 50443 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant decimal precision [doris]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4470339162
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant decimal precision [doris]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4470339154
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant decimal precision [doris]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4470383868
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant decimal precision [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #63322:
URL: https://github.com/apache/doris/pull/63322#discussion_r3254568675
##########
be/src/util/json/simd_json_parser.h:
##########
@@ -94,87 +106,234 @@ class SimdJSONParser {
}
private:
- simdjson::dom::array::iterator it;
+ std::vector<Node>::const_iterator it;
};
- ALWAYS_INLINE Array(const simdjson::dom::array& array_) : array(array_) {} /// NOLINT
- ALWAYS_INLINE Iterator begin() const { return array.begin(); }
- ALWAYS_INLINE Iterator end() const { return array.end(); }
- ALWAYS_INLINE size_t size() const { return array.size(); }
+ ALWAYS_INLINE explicit Array(const std::vector<Node>* array_) : array(array_) {} /// NOLINT
+ ALWAYS_INLINE Iterator begin() const { return Iterator(array->begin()); }
+ ALWAYS_INLINE Iterator end() const { return Iterator(array->end()); }
+ ALWAYS_INLINE size_t size() const { return array->size(); }
ALWAYS_INLINE Element operator[](size_t index) const {
assert(index < size());
- return array.at(index).value_unsafe();
+ return Element(&(*array)[index]);
}
private:
- simdjson::dom::array array;
+ const std::vector<Node>* array = nullptr;
};
using KeyValuePair = std::pair<std::string_view, Element>;
/// References an object in a JSON document.
class Object {
public:
class Iterator {
public:
- ALWAYS_INLINE Iterator(const simdjson::dom::object::iterator& it_)
- : it(it_) {} /// NOLINT
+ ALWAYS_INLINE explicit Iterator(const std::vector<std::string>* keys_,
+ const std::vector<Node>* values_, size_t index_)
+ : keys(keys_), values(values_), index(index_) {} /// NOLINT
ALWAYS_INLINE KeyValuePair operator*() const {
- const auto& res = *it;
- return {res.key, res.value};
+ return {(*keys)[index], Element(&(*values)[index])};
}
ALWAYS_INLINE Iterator& operator++() {
- ++it;
+ ++index;
return *this;
}
ALWAYS_INLINE Iterator operator++(int) {
auto res = *this;
- ++it;
+ ++index;
return res;
} /// NOLINT
ALWAYS_INLINE friend bool operator!=(const Iterator& left, const Iterator& right) {
- return left.it != right.it;
+ return left.index != right.index;
}
ALWAYS_INLINE friend bool operator==(const Iterator& left, const Iterator& right) {
return !(left != right);
}
private:
- simdjson::dom::object::iterator it;
+ const std::vector<std::string>* keys = nullptr;
+ const std::vector<Node>* values = nullptr;
+ size_t index = 0;
};
- ALWAYS_INLINE Object(const simdjson::dom::object& object_) : object(object_) {} /// NOLINT
- ALWAYS_INLINE Iterator begin() const { return object.begin(); }
- ALWAYS_INLINE Iterator end() const { return object.end(); }
- ALWAYS_INLINE size_t size() const { return object.size(); }
+ ALWAYS_INLINE explicit Object(const std::vector<std::string>* keys_,
+ const std::vector<Node>* values_)
+ : keys(keys_), values(values_) {} /// NOLINT
+ ALWAYS_INLINE Iterator begin() const { return Iterator(keys, values, 0); }
+ ALWAYS_INLINE Iterator end() const { return Iterator(keys, values, size()); }
+ ALWAYS_INLINE size_t size() const { return values->size(); }
/// Optional: Provides access to an object's element by index.
KeyValuePair operator[](size_t index) const {
assert(index < size());
- auto it = object.begin();
- while (index--) {
- ++it;
- }
- const auto& res = *it;
- return {res.key, res.value};
+ return {(*keys)[index], Element(&(*values)[index])};
}
private:
- simdjson::dom::object object;
+ const std::vector<std::string>* keys = nullptr;
+ const std::vector<Node>* values = nullptr;
};
/// Parses a JSON document, returns the reference to its root element if succeeded.
bool parse(const char* data, size_t size, Element& result) {
- auto document = parser.parse(data, size);
- if (document.error()) {
+ simdjson::padded_string padded_json(data, size);
+ simdjson::ondemand::document document;
+ auto error = parser.iterate(padded_json).get(document);
+ if (error) {
return false;
}
- result = document.value_unsafe();
+ root = Node();
+ if (!build_node(document, &root)) {
+ return false;
+ }
+ result = Element(&root);
return true;
}
private:
- simdjson::dom::parser parser;
+ template <typename RawNumber>
+ static bool assign_raw_number(RawNumber&& raw_number, std::string* out) {
+ if constexpr (std::is_same_v<std::decay_t<RawNumber>, std::string_view>) {
+ *out = std::string(raw_number);
+ return true;
+ } else {
+ std::string_view raw_number_view;
+ auto error = std::move(raw_number).get(raw_number_view);
+ if (error) {
+ return false;
+ }
+ *out = std::string(raw_number_view);
+ return true;
+ }
Review Comment:
This now eagerly materializes a second recursive JSON tree for every parse, copying every object key, string value, raw numeric token, and allocating vectors for every object/array before `JSONDataParser` traverses it. That happens even when `ParseConfig` has no decimal typed paths, so ordinary Variant ingestion pays a large CPU/memory cost and the allocations are in `std::string`/`std::vector` outside the existing column memory accounting. The raw-number preservation should stay on the hot path only when needed, or preserve raw numeric text without copying the whole document for untyped/non-decimal parses.
##########
be/src/util/json/simd_json_parser.h:
##########
@@ -94,87 +106,234 @@ class SimdJSONParser {
}
private:
- simdjson::dom::array::iterator it;
+ std::vector<Node>::const_iterator it;
};
- ALWAYS_INLINE Array(const simdjson::dom::array& array_) : array(array_) {} /// NOLINT
- ALWAYS_INLINE Iterator begin() const { return array.begin(); }
- ALWAYS_INLINE Iterator end() const { return array.end(); }
- ALWAYS_INLINE size_t size() const { return array.size(); }
+ ALWAYS_INLINE explicit Array(const std::vector<Node>* array_) : array(array_) {} /// NOLINT
+ ALWAYS_INLINE Iterator begin() const { return Iterator(array->begin()); }
+ ALWAYS_INLINE Iterator end() const { return Iterator(array->end()); }
+ ALWAYS_INLINE size_t size() const { return array->size(); }
ALWAYS_INLINE Element operator[](size_t index) const {
assert(index < size());
- return array.at(index).value_unsafe();
+ return Element(&(*array)[index]);
}
private:
- simdjson::dom::array array;
+ const std::vector<Node>* array = nullptr;
};
using KeyValuePair = std::pair<std::string_view, Element>;
/// References an object in a JSON document.
class Object {
public:
class Iterator {
public:
- ALWAYS_INLINE Iterator(const simdjson::dom::object::iterator& it_)
- : it(it_) {} /// NOLINT
+ ALWAYS_INLINE explicit Iterator(const std::vector<std::string>* keys_,
+ const std::vector<Node>* values_, size_t index_)
+ : keys(keys_), values(values_), index(index_) {} /// NOLINT
ALWAYS_INLINE KeyValuePair operator*() const {
- const auto& res = *it;
- return {res.key, res.value};
+ return {(*keys)[index], Element(&(*values)[index])};
}
ALWAYS_INLINE Iterator& operator++() {
- ++it;
+ ++index;
return *this;
}
ALWAYS_INLINE Iterator operator++(int) {
auto res = *this;
- ++it;
+ ++index;
return res;
} /// NOLINT
ALWAYS_INLINE friend bool operator!=(const Iterator& left, const Iterator& right) {
- return left.it != right.it;
+ return left.index != right.index;
}
ALWAYS_INLINE friend bool operator==(const Iterator& left, const Iterator& right) {
return !(left != right);
}
private:
- simdjson::dom::object::iterator it;
+ const std::vector<std::string>* keys = nullptr;
+ const std::vector<Node>* values = nullptr;
+ size_t index = 0;
};
- ALWAYS_INLINE Object(const simdjson::dom::object& object_) : object(object_) {} /// NOLINT
- ALWAYS_INLINE Iterator begin() const { return object.begin(); }
- ALWAYS_INLINE Iterator end() const { return object.end(); }
- ALWAYS_INLINE size_t size() const { return object.size(); }
+ ALWAYS_INLINE explicit Object(const std::vector<std::string>* keys_,
+ const std::vector<Node>* values_)
+ : keys(keys_), values(values_) {} /// NOLINT
+ ALWAYS_INLINE Iterator begin() const { return Iterator(keys, values, 0); }
+ ALWAYS_INLINE Iterator end() const { return Iterator(keys, values, size()); }
+ ALWAYS_INLINE size_t size() const { return values->size(); }
/// Optional: Provides access to an object's element by index.
KeyValuePair operator[](size_t index) const {
assert(index < size());
- auto it = object.begin();
- while (index--) {
- ++it;
- }
- const auto& res = *it;
- return {res.key, res.value};
+ return {(*keys)[index], Element(&(*values)[index])};
}
private:
- simdjson::dom::object object;
+ const std::vector<std::string>* keys = nullptr;
+ const std::vector<Node>* values = nullptr;
};
/// Parses a JSON document, returns the reference to its root element if succeeded.
bool parse(const char* data, size_t size, Element& result) {
- auto document = parser.parse(data, size);
- if (document.error()) {
+ simdjson::padded_string padded_json(data, size);
+ simdjson::ondemand::document document;
+ auto error = parser.iterate(padded_json).get(document);
+ if (error) {
return false;
}
- result = document.value_unsafe();
+ root = Node();
+ if (!build_node(document, &root)) {
+ return false;
+ }
+ result = Element(&root);
return true;
}
private:
- simdjson::dom::parser parser;
+ template <typename RawNumber>
+ static bool assign_raw_number(RawNumber&& raw_number, std::string* out) {
+ if constexpr (std::is_same_v<std::decay_t<RawNumber>, std::string_view>) {
+ *out = std::string(raw_number);
+ return true;
+ } else {
+ std::string_view raw_number_view;
+ auto error = std::move(raw_number).get(raw_number_view);
+ if (error) {
+ return false;
+ }
+ *out = std::string(raw_number_view);
+ return true;
+ }
+ }
+
+ template <typename Value>
+ bool build_array_node(Value& value, Node* out) {
+ simdjson::ondemand::array array;
+ auto error = value.get_array().get(array);
+ if (error) {
+ return false;
+ }
+ out->type = Node::Type::ARRAY;
+ for (auto element_result : array) {
+ simdjson::ondemand::value element;
+ error = std::move(element_result).get(element);
+ if (error) {
+ return false;
+ }
+ Node element_node;
+ if (!build_node(element, &element_node)) {
+ return false;
+ }
+ out->array_values.push_back(std::move(element_node));
+ }
+ return true;
+ }
+
+ template <typename Value>
+ bool build_object_node(Value& value, Node* out) {
+ simdjson::ondemand::object object;
+ auto error = value.get_object().get(object);
+ if (error) {
+ return false;
+ }
+ out->type = Node::Type::OBJECT;
+ for (auto field_result : object) {
+ simdjson::ondemand::field field;
+ error = std::move(field_result).get(field);
+ if (error) {
+ return false;
+ }
+ std::string_view key;
+ error = field.unescaped_key().get(key);
+ if (error) {
+ return false;
+ }
+ simdjson::ondemand::value field_value = field.value();
+ Node field_node;
+ if (!build_node(field_value, &field_node)) {
+ return false;
+ }
+ out->object_keys.emplace_back(key);
+ out->object_values.push_back(std::move(field_node));
+ }
+ return true;
+ }
+
+ template <typename Value>
+ bool build_number_node(Value& value, Node* out) {
+ if (!assign_raw_number(value.raw_json_token(), &out->raw_number)) {
+ return false;
+ }
+ simdjson::ondemand::number_type number_type;
+ auto error = value.get_number_type().get(number_type);
+ if (error) {
+ return false;
+ }
+ switch (number_type) {
+ case simdjson::ondemand::number_type::signed_integer:
+ out->type = Node::Type::INT64;
+ error = value.get_int64().get(out->int64_value);
+ return !error;
+ case simdjson::ondemand::number_type::unsigned_integer:
+ out->type = Node::Type::UINT64;
Review Comment:
`key` comes from `field.unescaped_key()`, whose returned view can point at simdjson's string buffer for escaped keys. This code does not copy it until after `build_node(field_value, ...)`, and building the value can parse another escaped string/key and reuse that buffer. For input such as `{"a\\nb": {"x": "escaped\\tvalue"}}`, the key view can be overwritten before `emplace_back(key)`, so the Variant path stored for the object field is corrupted or mismatched. Please copy `key` into `object_keys` before descending into the value, then append the value node, and add an escaped-key case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant decimal precision [doris]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4470508386
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant decimal precision [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4470943300
<details>
<summary>TPC-DS: <b>Total hot run time: 169036 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c34c703b31a24812923fba9197fe4db35efe1ccb, data reload: false
query5 4346 656 514 514
query6 339 222 193 193
query7 4240 567 320 320
query8 333 236 216 216
query9 8844 4090 4086 4086
query10 466 351 303 303
query11 5783 2354 2168 2168
query12 181 130 126 126
query13 1322 622 454 454
query14 6049 5391 5094 5094
query14_1 4440 4525 4363 4363
query15 219 206 182 182
query16 1016 452 497 452
query17 1164 729 605 605
query18 2497 499 375 375
query19 225 212 163 163
query20 138 136 131 131
query21 216 144 134 134
query22 13641 13608 13345 13345
query23 17342 16331 16028 16028
query23_1 16176 16101 16130 16101
query24 7352 1749 1280 1280
query24_1 1310 1307 1308 1307
query25 583 513 454 454
query26 1310 312 180 180
query27 2726 570 338 338
query28 4453 1964 1950 1950
query29 1019 649 511 511
query30 313 244 201 201
query31 1120 1071 954 954
query32 90 83 76 76
query33 573 361 297 297
query34 1154 1134 636 636
query35 767 785 674 674
query36 1326 1373 1118 1118
query37 165 114 90 90
query38 3216 3181 3021 3021
query39 929 929 903 903
query39_1 891 862 877 862
query40 235 146 123 123
query41 66 66 62 62
query42 109 108 107 107
query43 325 327 283 283
query44
query45 211 198 193 193
query46 1063 1150 737 737
query47 2264 2244 2195 2195
query48 399 397 297 297
query49 625 502 395 395
query50 972 350 255 255
query51 4410 4258 4228 4228
query52 105 105 98 98
query53 250 286 209 209
query54 317 285 254 254
query55 92 91 83 83
query56 295 321 303 303
query57 1394 1380 1315 1315
query58 293 266 264 264
query59 1554 1598 1447 1447
query60 318 329 315 315
query61 159 158 151 151
query62 684 633 569 569
query63 239 199 211 199
query64 2406 817 639 639
query65
query66 1739 471 369 369
query67 29910 29970 29801 29801
query68
query69 465 338 308 308
query70 970 981 1000 981
query71 307 281 269 269
query72 3092 2745 2387 2387
query73 805 788 414 414
query74 5072 4900 4738 4738
query75 2681 2588 2252 2252
query76 2308 1128 776 776
query77 397 408 332 332
query78 12125 12214 11606 11606
query79 1414 1061 722 722
query80 653 536 452 452
query81 454 277 243 243
query82 1378 154 127 127
query83 323 271 256 256
query84 266 141 110 110
query85 861 532 456 456
query86 404 348 311 311
query87 3400 3441 3221 3221
query88 3495 2671 2594 2594
query89 451 382 340 340
query90 1958 181 185 181
query91 175 168 139 139
query92 82 73 73 73
query93 1601 1418 879 879
query94 544 345 311 311
query95 690 396 343 343
query96 1045 804 312 312
query97 2671 2657 2529 2529
query98 245 225 234 225
query99 1116 1090 970 970
Total cold run time: 253132 ms
Total hot run time: 169036 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant decimal precision [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4471003543
<details>
<summary>TPC-DS: <b>Total hot run time: 169412 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 47d2d8e39a7486d4aec3c55d15f426a812f9df4c, data reload: false
query5 4326 648 514 514
query6 351 220 199 199
query7 4215 555 301 301
query8 337 236 222 222
query9 8836 3994 4006 3994
query10 455 343 300 300
query11 5791 2413 2196 2196
query12 188 132 157 132
query13 1273 612 414 414
query14 6047 5343 5033 5033
query14_1 4331 4332 4295 4295
query15 211 204 186 186
query16 992 449 416 416
query17 1136 716 594 594
query18 2450 486 348 348
query19 211 196 161 161
query20 138 135 131 131
query21 217 142 125 125
query22 13743 13596 13382 13382
query23 17262 16415 16018 16018
query23_1 16457 16432 16235 16235
query24 7524 1749 1300 1300
query24_1 1319 1307 1327 1307
query25 605 521 446 446
query26 1322 315 177 177
query27 2700 580 352 352
query28 4456 1984 1966 1966
query29 1267 632 516 516
query30 313 241 203 203
query31 1146 1072 945 945
query32 87 79 80 79
query33 560 372 303 303
query34 1194 1121 641 641
query35 767 794 682 682
query36 1341 1327 1155 1155
query37 157 111 96 96
query38 3209 3150 3031 3031
query39 930 938 913 913
query39_1 909 899 894 894
query40 232 153 127 127
query41 71 70 68 68
query42 115 114 110 110
query43 328 321 290 290
query44
query45 217 205 206 205
query46 1079 1240 752 752
query47 2303 2314 2142 2142
query48 418 419 310 310
query49 655 512 401 401
query50 1051 355 256 256
query51 4314 4319 4228 4228
query52 110 113 97 97
query53 260 287 213 213
query54 340 287 286 286
query55 99 95 88 88
query56 332 328 325 325
query57 1414 1397 1300 1300
query58 318 286 266 266
query59 1584 1630 1419 1419
query60 338 372 327 327
query61 149 157 153 153
query62 664 618 563 563
query63 243 206 210 206
query64 2467 791 648 648
query65
query66 1728 467 354 354
query67 30001 29896 29844 29844
query68
query69 451 349 308 308
query70 1003 1004 971 971
query71 315 283 270 270
query72 3003 2686 2406 2406
query73 860 783 431 431
query74 5065 4931 4708 4708
query75 2680 2607 2248 2248
query76 2279 1126 795 795
query77 407 426 352 352
query78 12224 12102 11681 11681
query79 1470 1039 726 726
query80 1324 574 456 456
query81 499 285 236 236
query82 1269 159 126 126
query83 346 272 259 259
query84 268 144 110 110
query85 926 542 445 445
query86 435 346 319 319
query87 3454 3332 3161 3161
query88 3542 2692 2646 2646
query89 459 386 331 331
query90 1803 181 181 181
query91 178 172 139 139
query92 78 75 73 73
query93 1490 1411 851 851
query94 681 352 302 302
query95 662 371 439 371
query96 1047 753 344 344
query97 2675 2670 2550 2550
query98 237 227 242 227
query99 1128 1123 968 968
Total cold run time: 255001 ms
Total hot run time: 169412 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant decimal precision [doris]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4471037422
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant decimal precision [doris]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4471159735
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant decimal precision [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4471340150
# BE UT Coverage Report
Increment line coverage `72.04% (304/422)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/6d77c2e151db0215a94add25f0ed9e0300fa3649_6d77c2e151db0215a94add25f0ed9e0300fa3649/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/6d77c2e151db0215a94add25f0ed9e0300fa3649_6d77c2e151db0215a94add25f0ed9e0300fa3649/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 53.51% (20659/38607) |
| Line Coverage | 37.17% (195304/525439) |
| Region Coverage | 33.55% (152875/455703) |
| Branch Coverage | 34.60% (66711/192829) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant decimal precision [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4471566814
<details>
<summary>TPC-DS: <b>Total hot run time: 169135 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 45d9060c080f192ecb190462a4b01b6485daad2a, data reload: false
query5 4345 633 524 524
query6 344 215 204 204
query7 4308 568 299 299
query8 325 230 220 220
query9 8828 4060 4025 4025
query10 478 351 311 311
query11 5832 2375 2237 2237
query12 187 132 129 129
query13 1320 625 419 419
query14 5986 5345 5039 5039
query14_1 4378 4355 4361 4355
query15 215 203 180 180
query16 995 439 456 439
query17 1164 751 598 598
query18 2727 483 356 356
query19 232 218 167 167
query20 152 136 131 131
query21 221 142 118 118
query22 13605 13473 13341 13341
query23 17244 16193 15908 15908
query23_1 16300 16343 16467 16343
query24 7897 1790 1338 1338
query24_1 1327 1357 1333 1333
query25 609 513 471 471
query26 1315 345 179 179
query27 2673 552 344 344
query28 4437 1990 1926 1926
query29 1028 644 516 516
query30 309 247 202 202
query31 1117 1068 940 940
query32 89 77 81 77
query33 536 356 294 294
query34 1166 1136 650 650
query35 761 785 711 711
query36 1317 1341 1248 1248
query37 154 105 91 91
query38 3186 3161 3061 3061
query39 921 909 902 902
query39_1 871 872 865 865
query40 231 148 124 124
query41 65 63 61 61
query42 112 108 105 105
query43 321 318 286 286
query44
query45 206 201 191 191
query46 1045 1209 718 718
query47 2321 2315 2180 2180
query48 401 405 274 274
query49 653 488 378 378
query50 940 357 257 257
query51 4296 4249 4194 4194
query52 105 105 92 92
query53 248 286 207 207
query54 307 264 247 247
query55 93 88 87 87
query56 318 312 322 312
query57 1403 1376 1293 1293
query58 291 270 255 255
query59 1554 1604 1406 1406
query60 313 322 300 300
query61 160 159 153 153
query62 658 631 556 556
query63 251 205 215 205
query64 2390 803 621 621
query65
query66 1673 473 344 344
query67 29963 29917 29797 29797
query68
query69 467 374 303 303
query70 983 998 958 958
query71 305 272 269 269
query72 3181 2637 2438 2438
query73 848 747 454 454
query74 5038 4890 4748 4748
query75 2668 2585 2250 2250
query76 2278 1131 757 757
query77 410 408 333 333
query78 12132 12020 11601 11601
query79 1486 1015 697 697
query80 1340 553 460 460
query81 524 286 238 238
query82 1026 153 119 119
query83 355 269 249 249
query84 263 142 114 114
query85 916 534 460 460
query86 450 334 313 313
query87 3469 3387 3199 3199
query88 3488 2677 2656 2656
query89 450 387 334 334
query90 1920 170 175 170
query91 178 177 145 145
query92 78 79 73 73
query93 1558 1397 848 848
query94 735 327 305 305
query95 695 390 446 390
query96 984 776 323 323
query97 2676 2667 2553 2553
query98 236 230 223 223
query99 1121 1080 981 981
Total cold run time: 254662 ms
Total hot run time: 169135 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant decimal precision [doris]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4470266101
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant decimal precision [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #63322:
URL: https://github.com/apache/doris/pull/63322#discussion_r3254422945
##########
be/src/util/json/json_parser.h:
##########
@@ -54,15 +93,19 @@ Field getValueAsField(const Element& element) {
return Field::create_field<TYPE_LARGEINT>(static_cast<int128_t>(element.getUInt64()));
}
if (element.isDouble()) {
+ if (auto raw_number = element.getRawNumber();
+ should_preserve_json_number_as_string(raw_number)) {
+ return Field::create_field<TYPE_STRING>(String(raw_number));
+ }
Review Comment:
This changes the parsed Field type for any JSON number that happens to have more than 15 significant digits or at least 10 fractional digits, even when the Variant path is not declared as a typed decimal path. For example, inserting `{"a": 1.2345678901}` into a normal `variant` column now stores/exposes `a` as `TYPE_STRING` instead of `TYPE_DOUBLE`, so `variant_type(v['a'])`, serialization, comparison behavior, and later least-common-type merging can change for existing untyped Variant data. The precision fix should preserve the raw token separately and only use it when casting to a decimal typed path, or otherwise keep the JSON number's Variant type numeric for untyped paths.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant predefined decimal precision [doris]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4474149471
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant predefined decimal precision [doris]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4474628800
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant predefined decimal precision [doris]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on code in PR #63322:
URL: https://github.com/apache/doris/pull/63322#discussion_r3256580937
##########
be/src/util/json/json_parser.cpp:
##########
@@ -84,8 +89,29 @@ void JSONDataParser<ParserImpl>::traverse(const Element& element, ParseContext&
// we should set has_nested_in_flatten to false when traverse array finished for next array otherwise it will be true for next array
ctx.has_nested_in_flatten = false;
} else {
- appendValueIfNotDuplicate(ctx, ctx.builder.get_parts(), getValueAsField(element));
+ appendValueIfNotDuplicate(ctx, ctx.builder.get_parts(),
+ getValueAsField(element, shouldPreserveNumberAsString(ctx)));
+ }
Review Comment:
这个线程描述的是 nested array/object 的 JSONB serialization path。CIR-20240 这次按最小修复范围只处理 Variant predefined decimal materialization 和 JSON reader 测试覆盖，不扩展到 JSONB；用户也明确要求 JSONB 先不处理，后续需要单独设计 schema-aware/JSONB 路径。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant predefined decimal precision [doris]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4474705089
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant predefined decimal precision [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4476853399
<details>
<summary>TPC-H: <b>Total hot run time: 31211 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 38f5b5def0cb7c4f587fcaad2726d59ac1f7ce4b, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17677 3828 3809 3809
q2 q3 10757 1390 808 808
q4 4680 479 352 352
q5 7589 2266 2149 2149
q6 248 180 140 140
q7 937 780 642 642
q8 9373 1785 1596 1596
q9 5138 4932 4909 4909
q10 6402 2096 1794 1794
q11 440 277 242 242
q12 633 429 288 288
q13 18138 3365 2833 2833
q14 268 253 235 235
q15 q16 822 777 708 708
q17 883 865 904 865
q18 7046 5913 5508 5508
q19 1176 1163 1074 1074
q20 510 405 264 264
q21 6290 2822 2669 2669
q22 467 393 326 326
Total cold run time: 99474 ms
Total hot run time: 31211 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4805 4577 4576 4576
q2 q3 4956 5317 4565 4565
q4 2138 2240 1417 1417
q5 4708 4890 4613 4613
q6 226 181 131 131
q7 1809 1764 1584 1584
q8 2403 2071 2059 2059
q9 7689 7455 7081 7081
q10 4486 4385 3931 3931
q11 523 379 347 347
q12 714 729 515 515
q13 2979 3408 2783 2783
q14 275 273 249 249
q15 q16 678 691 596 596
q17 1262 1239 1235 1235
q18 7227 6608 6766 6608
q19 1148 1073 1128 1073
q20 2211 2205 1934 1934
q21 5298 4637 4488 4488
q22 546 476 408 408
Total cold run time: 56081 ms
Total hot run time: 50193 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve Variant predefined decimal precision [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63322:
URL: https://github.com/apache/doris/pull/63322#issuecomment-4476938786
<details>
<summary>TPC-DS: <b>Total hot run time: 169559 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 38f5b5def0cb7c4f587fcaad2726d59ac1f7ce4b, data reload: false
query5 4323 640 511 511
query6 329 228 209 209
query7 4276 591 303 303
query8 321 234 223 223
query9 8838 3968 4002 3968
query10 442 336 295 295
query11 5867 2363 2213 2213
query12 181 128 121 121
query13 1284 613 439 439
query14 6050 5316 5009 5009
query14_1 4275 4307 4284 4284
query15 217 206 178 178
query16 990 449 451 449
query17 1112 723 594 594
query18 2456 491 348 348
query19 213 192 165 165
query20 140 130 127 127
query21 216 144 118 118
query22 13506 13532 13296 13296
query23 17357 16406 16032 16032
query23_1 16134 16086 16169 16086
query24 7482 1793 1287 1287
query24_1 1328 1285 1332 1285
query25 600 499 448 448
query26 1356 312 173 173
query27 2735 577 347 347
query28 4469 1936 1967 1936
query29 1014 655 553 553
query30 307 236 204 204
query31 1149 1054 951 951
query32 86 77 73 73
query33 550 364 306 306
query34 1178 1155 645 645
query35 789 796 678 678
query36 1353 1317 1224 1224
query37 158 110 97 97
query38 3217 3183 3065 3065
query39 935 942 909 909
query39_1 914 877 890 877
query40 232 155 128 128
query41 73 71 69 69
query42 114 113 118 113
query43 322 327 288 288
query44
query45 212 204 201 201
query46 1072 1193 725 725
query47 2347 2323 2247 2247
query48 432 421 314 314
query49 655 503 406 406
query50 1029 365 250 250
query51 4359 4347 4306 4306
query52 107 105 95 95
query53 254 278 206 206
query54 339 285 275 275
query55 94 92 90 90
query56 319 329 320 320
query57 1417 1418 1339 1339
query58 309 284 278 278
query59 1557 1637 1412 1412
query60 345 332 328 328
query61 210 149 150 149
query62 667 628 567 567
query63 240 195 202 195
query64 2452 815 642 642
query65
query66 1732 476 364 364
query67 30011 29945 29751 29751
query68
query69 470 343 305 305
query70 1027 980 993 980
query71 299 272 265 265
query72 3245 2711 2382 2382
query73 810 741 435 435
query74 5081 4905 4736 4736
query75 2677 2600 2252 2252
query76 2279 1151 767 767
query77 409 408 337 337
query78 12190 12164 11587 11587
query79 1477 1032 727 727
query80 647 536 451 451
query81 460 273 242 242
query82 1356 162 124 124
query83 358 277 249 249
query84 266 147 115 115
query85 889 549 464 464
query86 390 327 312 312
query87 3492 3361 3204 3204
query88 3523 2653 2641 2641
query89 454 385 340 340
query90 1964 182 174 174
query91 179 164 147 147
query92 79 80 76 76
query93 1412 1436 844 844
query94 553 326 304 304
query95 664 408 436 408
query96 1089 824 372 372
query97 2696 2697 2582 2582
query98 236 230 225 225
query99 1099 1114 1019 1019
Total cold run time: 253819 ms
Total hot run time: 169559 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org