You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "wsjz (via GitHub)" <gi...@apache.org> on 2024/03/25 07:21:33 UTC
[PR] [fix](insert)fix hive table sink type coercion and unify coercion [doris]
wsjz opened a new pull request, #32762:
URL: https://github.com/apache/doris/pull/32762
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](insert)fix hive table sink type coercion and unify coercion [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32762:
URL: https://github.com/apache/doris/pull/32762#issuecomment-2019608015
<details>
<summary>TPC-H: <b>Total hot run time: 37704 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 03c11715c0f4150e2181e1f97d7da191e7dd5a74, data reload: false
------ Round 1 ----------------------------------
q1 17617 4209 4067 4067
q2 2102 168 148 148
q3 10573 1112 1179 1112
q4 10229 775 783 775
q5 7469 2967 2976 2967
q6 202 127 124 124
q7 1024 586 580 580
q8 9337 1990 1975 1975
q9 7162 6569 6567 6567
q10 8426 3443 3566 3443
q11 435 228 222 222
q12 439 201 205 201
q13 17787 2850 2891 2850
q14 242 196 212 196
q15 500 469 450 450
q16 494 375 367 367
q17 947 611 518 518
q18 7083 6453 6421 6421
q19 1564 1364 1462 1364
q20 554 254 249 249
q21 3568 2919 2809 2809
q22 347 299 306 299
Total cold run time: 108101 ms
Total hot run time: 37704 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4040 4044 4045 4044
q2 332 234 236 234
q3 2948 2833 2817 2817
q4 1807 1566 1573 1566
q5 5308 5304 5299 5299
q6 194 115 119 115
q7 2258 1847 1863 1847
q8 3140 3290 3277 3277
q9 8696 8690 8664 8664
q10 3788 3765 3726 3726
q11 546 451 448 448
q12 715 544 563 544
q13 16920 2833 2841 2833
q14 282 256 259 256
q15 485 458 475 458
q16 469 451 416 416
q17 1710 1494 1453 1453
q18 7504 7272 7085 7085
q19 1618 1474 1532 1474
q20 1906 1718 1681 1681
q21 4663 4837 4579 4579
q22 546 457 458 457
Total cold run time: 69875 ms
Total hot run time: 53273 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](insert)fix hive table sink type coercion and unify coercion [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman merged PR #32762:
URL: https://github.com/apache/doris/pull/32762
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](insert)fix hive table sink type coercion and unify coercion [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #32762:
URL: https://github.com/apache/doris/pull/32762#issuecomment-2017586233
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](insert)fix hive table sink type coercion and unify coercion [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32762:
URL: https://github.com/apache/doris/pull/32762#issuecomment-2019739412
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit 03c11715c0f4150e2181e1f97d7da191e7dd5a74 with default session variables
Stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 21.9 seconds inserted 10000000 Rows, about 456K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](insert)fix hive table sink type coercion and unify coercion [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32762:
URL: https://github.com/apache/doris/pull/32762#issuecomment-2019712088
<details>
<summary>TPC-DS: <b>Total hot run time: 181216 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 03c11715c0f4150e2181e1f97d7da191e7dd5a74, data reload: false
query1 937 354 365 354
query2 6540 1873 1840 1840
query3 6708 210 211 210
query4 31702 21268 21271 21268
query5 4301 401 403 401
query6 279 178 174 174
query7 4620 285 303 285
query8 231 162 175 162
query9 9043 2354 2365 2354
query10 555 248 250 248
query11 17229 14125 14232 14125
query12 135 89 86 86
query13 1628 410 410 410
query14 10104 7956 7904 7904
query15 254 210 204 204
query16 8175 279 262 262
query17 1924 587 559 559
query18 2107 298 281 281
query19 301 157 156 156
query20 89 89 87 87
query21 207 134 130 130
query22 4971 4793 4730 4730
query23 33686 32631 32542 32542
query24 10722 2913 2810 2810
query25 580 389 399 389
query26 724 160 161 160
query27 2223 351 364 351
query28 5811 1872 1868 1868
query29 900 661 643 643
query30 307 152 150 150
query31 1019 761 714 714
query32 93 57 56 56
query33 663 257 248 248
query34 874 490 487 487
query35 828 619 618 618
query36 1015 890 882 882
query37 97 68 67 67
query38 3567 3466 3424 3424
query39 1462 1482 1417 1417
query40 208 128 123 123
query41 51 47 49 47
query42 104 97 94 94
query43 469 460 455 455
query44 1097 723 711 711
query45 272 272 268 268
query46 1118 692 685 685
query47 1901 1840 1843 1840
query48 430 385 350 350
query49 1088 340 338 338
query50 763 379 381 379
query51 6840 6855 6733 6733
query52 112 90 91 90
query53 351 277 279 277
query54 301 233 231 231
query55 84 80 83 80
query56 257 232 227 227
query57 1210 1150 1130 1130
query58 237 211 215 211
query59 2836 2594 2674 2594
query60 262 247 247 247
query61 114 112 110 110
query62 675 454 463 454
query63 309 274 278 274
query64 5456 4098 4033 4033
query65 3064 3003 2989 2989
query66 890 367 351 351
query67 15220 15018 14548 14548
query68 7667 524 527 524
query69 636 379 368 368
query70 1215 1135 1092 1092
query71 487 261 265 261
query72 6481 2728 2541 2541
query73 727 320 334 320
query74 7740 6394 6407 6394
query75 3946 2238 2201 2201
query76 4888 857 881 857
query77 655 256 255 255
query78 10850 10278 10054 10054
query79 10152 512 516 512
query80 1777 396 374 374
query81 542 211 222 211
query82 846 90 90 90
query83 225 143 141 141
query84 289 77 78 77
query85 1337 325 305 305
query86 468 274 293 274
query87 3727 3551 3545 3545
query88 4940 2379 2375 2375
query89 488 374 366 366
query90 1941 179 179 179
query91 168 135 133 133
query92 59 48 47 47
query93 6952 504 488 488
query94 1171 177 184 177
query95 449 333 334 333
query96 610 265 275 265
query97 2674 2502 2476 2476
query98 237 212 207 207
query99 1251 931 849 849
Total cold run time: 307925 ms
Total hot run time: 181216 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](insert)fix hive table sink type coercion and unify coercion [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #32762:
URL: https://github.com/apache/doris/pull/32762#issuecomment-2018379971
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](insert)fix hive table sink type coercion and unify coercion [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #32762:
URL: https://github.com/apache/doris/pull/32762#issuecomment-2017587180
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](insert)fix hive table sink type coercion and unify coercion [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #32762:
URL: https://github.com/apache/doris/pull/32762#issuecomment-2017587233
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](insert)fix hive table sink type coercion and unify coercion [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32762:
URL: https://github.com/apache/doris/pull/32762#issuecomment-2017674102
<details>
<summary>TPC-DS: <b>Total hot run time: 185893 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7910bd66fa5282657b59e7b7d551fa4f5fdcc672, data reload: false
query1 964 369 345 345
query2 7356 2016 1927 1927
query3 6709 212 208 208
query4 31561 21215 21395 21215
query5 4358 394 497 394
query6 266 171 181 171
query7 4624 282 285 282
query8 231 169 171 169
query9 9452 2245 2253 2245
query10 562 243 252 243
query11 15430 14285 14427 14285
query12 140 96 85 85
query13 1615 416 415 415
query14 11776 10715 10942 10715
query15 277 195 192 192
query16 8107 259 254 254
query17 1946 583 557 557
query18 2041 291 282 282
query19 246 152 150 150
query20 95 90 87 87
query21 202 128 127 127
query22 5005 4854 4793 4793
query23 33190 32863 32734 32734
query24 10840 2843 2911 2843
query25 596 386 395 386
query26 815 160 155 155
query27 2355 344 349 344
query28 6250 1879 1827 1827
query29 883 648 612 612
query30 295 154 142 142
query31 939 722 723 722
query32 91 53 55 53
query33 758 312 255 255
query34 1014 459 485 459
query35 833 595 594 594
query36 1008 902 900 900
query37 103 74 72 72
query38 3533 3460 3443 3443
query39 1484 1433 1432 1432
query40 199 110 107 107
query41 50 44 46 44
query42 103 94 97 94
query43 472 450 445 445
query44 1185 710 700 700
query45 278 256 258 256
query46 1118 706 718 706
query47 1933 1820 1830 1820
query48 432 356 339 339
query49 1098 340 329 329
query50 764 369 365 365
query51 6761 6610 6626 6610
query52 104 84 88 84
query53 341 275 274 274
query54 305 227 243 227
query55 81 82 77 77
query56 256 217 216 216
query57 1190 1128 1132 1128
query58 225 201 203 201
query59 2798 2575 2604 2575
query60 260 256 327 256
query61 97 93 96 93
query62 725 450 454 450
query63 306 279 274 274
query64 4920 4174 3977 3977
query65 3057 3009 3015 3009
query66 883 393 375 375
query67 15226 14719 14827 14719
query68 5274 521 513 513
query69 574 378 375 375
query70 1220 1200 1128 1128
query71 432 281 302 281
query72 6456 2850 2660 2660
query73 718 320 310 310
query74 7010 6772 6660 6660
query75 3516 2889 2840 2840
query76 3820 873 847 847
query77 633 276 273 273
query78 11068 10160 10189 10160
query79 8367 512 511 511
query80 1829 410 385 385
query81 550 212 218 212
query82 1600 212 211 211
query83 282 141 141 141
query84 288 81 74 74
query85 1501 322 311 311
query86 496 313 324 313
query87 3727 3577 3558 3558
query88 5370 2289 2290 2289
query89 533 368 378 368
query90 1944 174 176 174
query91 174 136 137 136
query92 64 47 48 47
query93 6664 501 485 485
query94 1213 173 174 173
query95 431 337 327 327
query96 613 269 272 269
query97 3051 2877 2896 2877
query98 229 221 217 217
query99 1198 904 917 904
Total cold run time: 303962 ms
Total hot run time: 185893 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](insert)fix hive table sink type coercion and unify coercion [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32762:
URL: https://github.com/apache/doris/pull/32762#issuecomment-2017679888
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit 7910bd66fa5282657b59e7b7d551fa4f5fdcc672 with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select: 21.2 seconds inserted 10000000 Rows, about 471K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](insert)fix hive table sink type coercion and unify coercion [doris]
Posted by "wsjz (via GitHub)" <gi...@apache.org>.
wsjz commented on PR #32762:
URL: https://github.com/apache/doris/pull/32762#issuecomment-2019539533
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](insert)fix hive table sink type coercion and unify coercion [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32762:
URL: https://github.com/apache/doris/pull/32762#issuecomment-2017371829
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website).
See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](insert)fix hive table sink type coercion and unify coercion [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32762:
URL: https://github.com/apache/doris/pull/32762#issuecomment-2017652406
<details>
<summary>TPC-H: <b>Total hot run time: 37822 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7910bd66fa5282657b59e7b7d551fa4f5fdcc672, data reload: false
------ Round 1 ----------------------------------
q1 17635 4256 4059 4059
q2 2105 159 151 151
q3 10593 1178 1189 1178
q4 10234 748 835 748
q5 7536 3018 2971 2971
q6 200 123 120 120
q7 1022 585 563 563
q8 9329 2017 1979 1979
q9 7140 6634 6552 6552
q10 8475 3439 3551 3439
q11 432 216 220 216
q12 413 198 197 197
q13 17799 2870 2857 2857
q14 231 205 204 204
q15 511 459 451 451
q16 498 373 372 372
q17 937 536 592 536
q18 7105 6542 6417 6417
q19 1919 1437 1463 1437
q20 576 276 285 276
q21 3501 2880 2816 2816
q22 331 285 283 283
Total cold run time: 108522 ms
Total hot run time: 37822 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4086 4067 4043 4043
q2 321 220 227 220
q3 2937 2877 2829 2829
q4 1818 1511 1510 1510
q5 5297 5343 5329 5329
q6 193 115 113 113
q7 2214 1843 1828 1828
q8 3141 3266 3246 3246
q9 8671 8677 8621 8621
q10 3799 3758 3722 3722
q11 532 447 438 438
q12 722 543 564 543
q13 16897 2892 2857 2857
q14 276 248 243 243
q15 488 464 465 464
q16 477 425 419 419
q17 1723 1498 1470 1470
q18 7497 7133 7183 7133
q19 1615 1547 1536 1536
q20 1934 1731 1729 1729
q21 4909 4748 4827 4748
q22 518 446 457 446
Total cold run time: 70065 ms
Total hot run time: 53487 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org