You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "morningman (via GitHub)" <gi...@apache.org> on 2023/11/23 05:49:02 UTC
[PR] [deprecated](external) remove deprecated hudi and iceberg external table [doris]
morningman opened a new pull request, #27456:
URL: https://github.com/apache/doris/pull/27456
The creation of hudi and iceberg table is disallowed since v1.2.
All these features are covered by hudi/iceberg catalog.
We should remove the code in v2.1
The PR mainly changes:
1. remove the code of hudi/iceberg external table.
2. remove code of iceberg database.
3. disallowed hive external table's creation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [deprecated](external) remove deprecated hudi and iceberg external table [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27456:
URL: https://github.com/apache/doris/pull/27456#issuecomment-1825256552
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit 845246c202d9a730684088c8c9f4dbf13993b555, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4916 4725 4684 4684
q2 357 165 151 151
q3 2044 1887 1892 1887
q4 1407 1279 1257 1257
q5 3973 3942 4046 3942
q6 257 136 133 133
q7 1443 877 892 877
q8 2799 2799 2780 2780
q9 9754 9601 9447 9447
q10 3478 3532 3533 3532
q11 385 245 242 242
q12 445 301 307 301
q13 4583 3860 3842 3842
q14 322 293 294 293
q15 591 544 520 520
q16 667 583 592 583
q17 1137 968 937 937
q18 7785 7296 7383 7296
q19 1674 1701 1686 1686
q20 581 306 330 306
q21 4479 3990 4057 3990
q22 476 370 370 370
Total cold run time: 53553 ms
Total hot run time: 49056 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4604 4600 4559 4559
q2 350 234 251 234
q3 4004 3991 4010 3991
q4 2711 2707 2710 2707
q5 9547 9633 9575 9575
q6 254 123 130 123
q7 3017 2511 2491 2491
q8 4437 4405 4389 4389
q9 12903 12880 12786 12786
q10 4073 4157 4171 4157
q11 783 660 663 660
q12 982 829 837 829
q13 4295 3598 3584 3584
q14 382 345 355 345
q15 573 519 530 519
q16 731 678 675 675
q17 3891 3851 3852 3851
q18 9434 8868 8970 8868
q19 1852 1787 1795 1787
q20 2387 2068 2057 2057
q21 8798 8699 8703 8699
q22 888 814 839 814
Total cold run time: 80896 ms
Total hot run time: 77700 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [deprecated](external) remove deprecated hudi and iceberg external table [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27456:
URL: https://github.com/apache/doris/pull/27456#issuecomment-1823967055
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit db61aff34b6ee48c56658f850a0f51e5c2f3fb71, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4899 4726 4664 4664
q2 357 154 145 145
q3 2045 1928 1918 1918
q4 1394 1282 1241 1241
q5 3961 3985 4058 3985
q6 256 131 132 131
q7 1437 897 882 882
q8 2790 2828 2788 2788
q9 9629 9439 9481 9439
q10 3483 3525 3494 3494
q11 382 249 250 249
q12 441 293 298 293
q13 4559 3803 3799 3799
q14 313 288 298 288
q15 586 545 520 520
q16 666 581 584 581
q17 1145 989 941 941
q18 7781 7435 7424 7424
q19 1693 1708 1704 1704
q20 561 311 305 305
q21 4434 3991 3985 3985
q22 472 379 378 378
Total cold run time: 53284 ms
Total hot run time: 49154 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4636 4611 4558 4558
q2 382 250 284 250
q3 4028 3998 3992 3992
q4 2718 2699 2700 2699
q5 9746 9685 9656 9656
q6 250 123 124 123
q7 3035 2503 2507 2503
q8 4434 4435 4406 4406
q9 12977 12947 12891 12891
q10 4093 4192 4202 4192
q11 784 719 633 633
q12 971 819 813 813
q13 4311 3556 3618 3556
q14 393 362 345 345
q15 579 517 539 517
q16 739 695 671 671
q17 3977 3816 3969 3816
q18 9742 9099 9117 9099
q19 1813 1796 1814 1796
q20 2386 2081 2084 2081
q21 8940 8850 8835 8835
q22 896 789 835 789
Total cold run time: 81830 ms
Total hot run time: 78221 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [deprecated](external) remove deprecated hudi and iceberg external table [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27456:
URL: https://github.com/apache/doris/pull/27456#issuecomment-1825266510
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.51 seconds
stream load tsv: 581 seconds loaded 74807831229 Bytes, about 122 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 27.9 seconds inserted 10000000 Rows, about 358K ops/s
storage size: 17101819074 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [deprecated](external) remove deprecated hudi and iceberg external table [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #27456:
URL: https://github.com/apache/doris/pull/27456#issuecomment-1823868335
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [deprecated](external) remove deprecated hudi and iceberg external table [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #27456:
URL: https://github.com/apache/doris/pull/27456#issuecomment-1824278465
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [deprecated](external) remove deprecated hudi and iceberg external table [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27456:
URL: https://github.com/apache/doris/pull/27456#issuecomment-1824769126
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.68 seconds
stream load tsv: 566 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.6 seconds inserted 10000000 Rows, about 337K ops/s
storage size: 17100753669 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [deprecated](external) remove deprecated hudi and iceberg external table [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27456:
URL: https://github.com/apache/doris/pull/27456#issuecomment-1824476491
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit 18498ee015da703a4b36590da6150465b53ce907, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4922 4638 4659 4638
q2 360 149 138 138
q3 2061 1994 1892 1892
q4 1393 1288 1266 1266
q5 3955 3950 4011 3950
q6 251 128 130 128
q7 1442 886 880 880
q8 2797 2796 2804 2796
q9 9769 9451 9337 9337
q10 3443 3530 3511 3511
q11 385 244 242 242
q12 436 291 299 291
q13 4613 3831 3800 3800
q14 314 302 287 287
q15 592 536 520 520
q16 666 587 594 587
q17 1136 957 917 917
q18 7774 7329 7330 7329
q19 1679 1705 1684 1684
q20 562 315 292 292
q21 4431 3982 3990 3982
q22 474 382 380 380
Total cold run time: 53455 ms
Total hot run time: 48847 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4585 4584 4562 4562
q2 352 248 258 248
q3 4039 4012 4004 4004
q4 2722 2696 2694 2694
q5 9697 9700 9673 9673
q6 251 125 125 125
q7 3031 2505 2485 2485
q8 4444 4468 4442 4442
q9 12943 12974 12891 12891
q10 4065 4168 4148 4148
q11 817 651 666 651
q12 981 816 825 816
q13 4293 3613 3547 3547
q14 382 353 342 342
q15 563 523 525 523
q16 753 678 686 678
q17 3906 3974 3898 3898
q18 9525 9097 9099 9097
q19 1857 1781 1777 1777
q20 2395 2040 2066 2040
q21 8848 8639 8711 8639
q22 919 803 849 803
Total cold run time: 81368 ms
Total hot run time: 78083 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [deprecated](external) remove deprecated hudi and iceberg external table [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman merged PR #27456:
URL: https://github.com/apache/doris/pull/27456
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [deprecated](external) remove deprecated hudi and iceberg external table [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27456:
URL: https://github.com/apache/doris/pull/27456#issuecomment-1824760653
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit 7c2f411644b0ed30a74110352ac8380cd97b12f9, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4943 4649 4647 4647
q2 379 176 159 159
q3 2046 1923 1878 1878
q4 1392 1275 1307 1275
q5 3958 3955 4015 3955
q6 252 130 127 127
q7 1456 895 873 873
q8 2787 2819 2799 2799
q9 45289 13610 9465 9465
q10 12873 3535 3525 3525
q11 397 250 237 237
q12 1050 288 286 286
q13 19739 3799 3750 3750
q14 315 289 278 278
q15 583 527 523 523
q16 681 594 579 579
q17 1163 991 962 962
q18 7820 7410 7314 7314
q19 1714 1672 1680 1672
q20 555 296 304 296
q21 6190 4001 4006 4001
q22 483 362 373 362
Total cold run time: 116065 ms
Total hot run time: 48963 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4634 4568 4596 4568
q2 341 227 265 227
q3 4009 4005 3996 3996
q4 2709 2703 2698 2698
q5 9649 9608 9552 9552
q6 240 128 121 121
q7 3016 2469 2424 2424
q8 4482 4418 4448 4418
q9 12860 12732 12744 12732
q10 4103 4170 4190 4170
q11 730 638 646 638
q12 980 828 818 818
q13 4282 3571 3533 3533
q14 385 344 343 343
q15 569 527 517 517
q16 729 673 671 671
q17 3863 3890 3873 3873
q18 9525 8949 9115 8949
q19 1835 1807 1787 1787
q20 2405 2047 2054 2047
q21 8818 8881 8579 8579
q22 881 794 825 794
Total cold run time: 81045 ms
Total hot run time: 77455 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [deprecated](external) remove deprecated hudi and iceberg external table [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27456:
URL: https://github.com/apache/doris/pull/27456#issuecomment-1825659141
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [deprecated](external) remove deprecated hudi and iceberg external table [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #27456:
URL: https://github.com/apache/doris/pull/27456#issuecomment-1825176304
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [deprecated](external) remove deprecated hudi and iceberg external table [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27456:
URL: https://github.com/apache/doris/pull/27456#issuecomment-1824471240
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 43.49 seconds
stream load tsv: 577 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.1 seconds inserted 10000000 Rows, about 355K ops/s
storage size: 17100578900 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [deprecated](external) remove deprecated hudi and iceberg external table [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27456:
URL: https://github.com/apache/doris/pull/27456#issuecomment-1823970414
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 43.66 seconds
stream load tsv: 583 seconds loaded 74807831229 Bytes, about 122 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.5 seconds inserted 10000000 Rows, about 350K ops/s
storage size: 17100689296 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [deprecated](external) remove deprecated hudi and iceberg external table [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #27456:
URL: https://github.com/apache/doris/pull/27456#issuecomment-1824595296
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [deprecated](external) remove deprecated hudi and iceberg external table [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27456:
URL: https://github.com/apache/doris/pull/27456#issuecomment-1825659186
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org