You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "dutyu (via GitHub)" <gi...@apache.org> on 2023/12/17 08:47:14 UTC
[PR] [feature-wip](nereids) Support some spark-sqll built-in functions when set dialect=spark_sql [doris]
dutyu opened a new pull request, #28531:
URL: https://github.com/apache/doris/pull/28531
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature-wip](nereids) Support some spark-sql built-in functions when set dialect=spark_sql [doris]
Posted by "dutyu (via GitHub)" <gi...@apache.org>.
dutyu commented on PR #28531:
URL: https://github.com/apache/doris/pull/28531#issuecomment-1860076793
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature-wip](nereids) Support some spark-sql built-in functions when set dialect=spark_sql [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28531:
URL: https://github.com/apache/doris/pull/28531#issuecomment-1863808203
TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Tpch sf100 test result on commit 0caae937a754d04fb67aaf8d0ecda7f99c9477a1, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4727 4457 4430 4430
q2 369 146 158 146
q3 1481 1254 1233 1233
q4 1119 963 901 901
q5 3149 3162 3174 3162
q6 270 129 131 129
q7 999 488 513 488
q8 2196 2214 2195 2195
q9 6680 6678 6692 6678
q10 3232 3274 3295 3274
q11 315 189 182 182
q12 357 222 218 218
q13 4592 3817 3802 3802
q14 245 218 219 218
q15 576 526 534 526
q16 447 403 395 395
q17 999 553 590 553
q18 7153 6975 7006 6975
q19 1517 1392 1486 1392
q20 537 310 288 288
q21 3044 2653 2672 2653
q22 358 296 296 296
Total cold run time: 44362 ms
Total hot run time: 40134 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4425 4379 4351 4351
q2 272 167 174 167
q3 3557 3531 3524 3524
q4 2394 2389 2385 2385
q5 5776 5771 5739 5739
q6 242 120 124 120
q7 2416 1876 1880 1876
q8 3531 3511 3529 3511
q9 9128 8979 8994 8979
q10 3949 4017 4005 4005
q11 496 384 386 384
q12 779 614 619 614
q13 4332 3560 3588 3560
q14 280 264 261 261
q15 566 519 517 517
q16 512 463 494 463
q17 1888 1882 1842 1842
q18 8621 8350 8196 8196
q19 1790 1744 1749 1744
q20 2265 1940 1967 1940
q21 6552 6233 6182 6182
q22 519 447 439 439
Total cold run time: 64290 ms
Total hot run time: 60799 ms
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature-wip](nereids) Support some spark-sql built-in functions when set dialect=spark_sql [doris]
Posted by "dutyu (via GitHub)" <gi...@apache.org>.
dutyu commented on PR #28531:
URL: https://github.com/apache/doris/pull/28531#issuecomment-1863772831
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature-wip](nereids) Support some spark-sql built-in functions when set dialect=spark_sql [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #28531:
URL: https://github.com/apache/doris/pull/28531#issuecomment-1871893281
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature-wip](nereids) Support some spark-sql built-in functions when set dialect=spark_sql [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28531:
URL: https://github.com/apache/doris/pull/28531#issuecomment-1860306478
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 43.67 seconds
stream load tsv: 571 seconds loaded 74807831229 Bytes, about 124 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17219673593 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature-wip](nereids) Support some spark-sqll built-in functions when set dialect=spark_sql [doris]
Posted by "dutyu (via GitHub)" <gi...@apache.org>.
dutyu commented on PR #28531:
URL: https://github.com/apache/doris/pull/28531#issuecomment-1859422251
run compile
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature-wip](nereids) Support some spark-sql built-in functions when set dialect=spark_sql [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28531:
URL: https://github.com/apache/doris/pull/28531#issuecomment-1860274028
TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Tpch sf100 test result on commit 6be5d78473ff8dd44901841cb88038c909793182, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4697 4476 4477 4476
q2 380 173 160 160
q3 1458 1255 1235 1235
q4 1119 893 909 893
q5 3147 3138 3168 3138
q6 242 128 128 128
q7 977 493 489 489
q8 2229 2238 2205 2205
q9 6721 6667 6670 6667
q10 3236 3250 3251 3250
q11 327 210 200 200
q12 353 207 204 204
q13 4578 3819 3809 3809
q14 239 222 211 211
q15 571 523 531 523
q16 445 390 382 382
q17 1011 581 528 528
q18 7212 6818 6849 6818
q19 1536 1442 1401 1401
q20 561 325 301 301
q21 3103 2627 2691 2627
q22 349 278 289 278
Total cold run time: 44491 ms
Total hot run time: 39923 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4395 4368 4408 4368
q2 266 165 171 165
q3 3539 3530 3531 3530
q4 2394 2385 2391 2385
q5 5769 5725 5697 5697
q6 244 123 120 120
q7 2389 1885 1844 1844
q8 3521 3522 3516 3516
q9 9005 8976 8953 8953
q10 3918 3988 3974 3974
q11 508 383 387 383
q12 765 592 584 584
q13 4287 3542 3551 3542
q14 284 254 267 254
q15 571 522 525 522
q16 531 449 467 449
q17 1890 1861 1824 1824
q18 8597 8205 8275 8205
q19 1739 1771 1730 1730
q20 2252 1948 1935 1935
q21 6534 6166 6205 6166
q22 494 437 434 434
Total cold run time: 63892 ms
Total hot run time: 60580 ms
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature-wip](nereids) Support some spark-sql built-in functions when set dialect=spark_sql [doris]
Posted by "dutyu (via GitHub)" <gi...@apache.org>.
dutyu commented on PR #28531:
URL: https://github.com/apache/doris/pull/28531#issuecomment-1862089771
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature-wip](nereids) Support some spark-sql built-in functions when set dialect=spark_sql [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28531:
URL: https://github.com/apache/doris/pull/28531#issuecomment-1859506218
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 43.75 seconds
stream load tsv: 588 seconds loaded 74807831229 Bytes, about 121 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 67 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17222435616 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature-wip](nereids) Support some spark-sql built-in functions when set dialect=spark_sql [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #28531:
URL: https://github.com/apache/doris/pull/28531#issuecomment-1871893254
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature-wip](nereids) Support some spark-sql built-in functions when set dialect=spark_sql [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28531:
URL: https://github.com/apache/doris/pull/28531#issuecomment-1862126843
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.66 seconds
stream load tsv: 577 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17222652727 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature-wip](nereids) Support some spark-sql built-in functions when set dialect=spark_sql [doris]
Posted by "dutyu (via GitHub)" <gi...@apache.org>.
dutyu commented on PR #28531:
URL: https://github.com/apache/doris/pull/28531#issuecomment-1859485087
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature-wip](nereids) Support some spark-sqll built-in functions when set dialect=spark_sql [doris]
Posted by "dutyu (via GitHub)" <gi...@apache.org>.
dutyu commented on PR #28531:
URL: https://github.com/apache/doris/pull/28531#issuecomment-1859076236
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature-wip](nereids) Support some spark-sql built-in functions when set dialect=spark_sql [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28531:
URL: https://github.com/apache/doris/pull/28531#issuecomment-1859505327
TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Tpch sf100 test result on commit 8a1312a463ac43fc9848126a41328e61ca5db044, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4705 4445 4459 4445
q2 370 146 158 146
q3 1454 1249 1209 1209
q4 1110 915 914 914
q5 3138 3139 3153 3139
q6 244 125 126 125
q7 990 481 492 481
q8 2178 2213 2179 2179
q9 6669 6643 6687 6643
q10 3218 3271 3254 3254
q11 330 211 207 207
q12 354 209 207 207
q13 4551 3815 3820 3815
q14 244 211 215 211
q15 566 533 523 523
q16 434 385 393 385
q17 999 595 517 517
q18 7209 6848 6918 6848
q19 1524 1453 1363 1363
q20 545 302 321 302
q21 3088 2697 2702 2697
q22 343 286 289 286
Total cold run time: 44263 ms
Total hot run time: 39896 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4380 4371 4368 4368
q2 266 161 172 161
q3 3526 3517 3513 3513
q4 2400 2373 2381 2373
q5 5738 5735 5706 5706
q6 244 118 123 118
q7 2365 1897 1841 1841
q8 3522 3511 3510 3510
q9 9029 8982 9006 8982
q10 3917 3989 3985 3985
q11 500 383 369 369
q12 768 587 601 587
q13 4305 3596 3579 3579
q14 282 244 260 244
q15 581 522 522 522
q16 511 463 478 463
q17 1877 1838 1824 1824
q18 8701 8321 8236 8236
q19 1742 1745 1762 1745
q20 2257 1957 1956 1956
q21 6559 6184 6164 6164
q22 495 421 420 420
Total cold run time: 63965 ms
Total hot run time: 60666 ms
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature-wip](nereids) Support some spark-sql built-in functions when set dialect=spark_sql [doris]
Posted by "dutyu (via GitHub)" <gi...@apache.org>.
dutyu commented on PR #28531:
URL: https://github.com/apache/doris/pull/28531#issuecomment-1860211294
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature-wip](nereids) Support some spark-sql built-in functions when set dialect=spark_sql [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28531:
URL: https://github.com/apache/doris/pull/28531#issuecomment-1862121686
TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Tpch sf100 test result on commit c1d952385405cc2b77688cd7f7f70a7230ae1181, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4693 4419 4417 4417
q2 367 147 159 147
q3 1472 1282 1245 1245
q4 1118 906 881 881
q5 3152 3155 3161 3155
q6 249 132 133 132
q7 985 492 479 479
q8 2200 2233 2192 2192
q9 6685 6701 6667 6667
q10 3227 3266 3268 3266
q11 307 188 180 180
q12 357 208 207 207
q13 4521 3813 3839 3813
q14 241 214 212 212
q15 557 526 520 520
q16 447 381 388 381
q17 1018 552 507 507
q18 7198 7003 6864 6864
q19 1542 1420 1453 1420
q20 550 299 282 282
q21 3094 2666 2652 2652
q22 341 283 283 283
Total cold run time: 44321 ms
Total hot run time: 39902 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4361 4359 4297 4297
q2 266 164 169 164
q3 3536 3520 3516 3516
q4 2376 2362 2361 2361
q5 5717 5714 5742 5714
q6 244 119 126 119
q7 2380 1901 1882 1882
q8 3537 3543 3525 3525
q9 9015 9059 9018 9018
q10 3924 3982 4005 3982
q11 483 360 356 356
q12 775 591 592 591
q13 4257 3565 3570 3565
q14 297 263 264 263
q15 567 536 521 521
q16 501 455 471 455
q17 1888 1849 1824 1824
q18 8651 8261 8334 8261
q19 1725 1751 1733 1733
q20 2249 1946 1931 1931
q21 6455 6170 6149 6149
q22 491 422 453 422
Total cold run time: 63695 ms
Total hot run time: 60649 ms
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature-wip](nereids) Support some spark-sql built-in functions when set dialect=spark_sql [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei merged PR #28531:
URL: https://github.com/apache/doris/pull/28531
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org