You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "suxiaogang223 (via GitHub)" <gi...@apache.org> on 2025/05/13 15:47:24 UTC
[PR] [fix](hive) fix write hive partition by Doris [doris]
suxiaogang223 opened a new pull request, #50864:
URL: https://github.com/apache/doris/pull/50864
### What problem does this PR solve?
Problem Summary:
After writing to a Hive partitioned table and adding a new partition using Doris, writing data to the same partition using Hive results in an error: “Partition column xxx conflicts with table columns.”
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](hive) fix write hive partition by Doris [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #50864:
URL: https://github.com/apache/doris/pull/50864#issuecomment-2877254717
<details>
<summary>TPC-DS: <b>Total hot run time: 193979 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit acaed5e681e8399e30f8cbbe396fb3a9072336c5, data reload: false
query1 1370 1110 1069 1069
query2 6115 1880 1860 1860
query3 10983 4276 4452 4276
query4 52935 25436 23402 23402
query5 4964 583 444 444
query6 329 206 203 203
query7 4895 522 291 291
query8 316 253 240 240
query9 5284 2651 2647 2647
query10 425 320 263 263
query11 15124 14974 14764 14764
query12 160 108 100 100
query13 1033 532 437 437
query14 10163 6341 6197 6197
query15 191 207 180 180
query16 7058 697 488 488
query17 1112 751 581 581
query18 1551 425 318 318
query19 214 186 171 171
query20 127 124 124 124
query21 208 126 118 118
query22 4753 4974 4863 4863
query23 34387 33520 33734 33520
query24 6603 2445 2442 2442
query25 462 472 396 396
query26 735 278 150 150
query27 2411 521 345 345
query28 3842 2176 2159 2159
query29 603 590 436 436
query30 268 229 186 186
query31 860 863 788 788
query32 73 65 64 64
query33 450 367 307 307
query34 796 873 528 528
query35 784 834 787 787
query36 931 992 882 882
query37 127 107 82 82
query38 4290 4244 4231 4231
query39 1516 1504 1488 1488
query40 227 122 107 107
query41 55 53 56 53
query42 134 111 120 111
query43 524 523 501 501
query44 1336 843 819 819
query45 179 172 165 165
query46 882 1041 644 644
query47 1886 1865 1834 1834
query48 413 441 335 335
query49 690 540 454 454
query50 717 741 409 409
query51 4203 4224 4199 4199
query52 117 112 100 100
query53 231 268 184 184
query54 618 597 525 525
query55 84 82 82 82
query56 315 313 292 292
query57 1163 1183 1154 1154
query58 266 268 255 255
query59 2826 2955 2813 2813
query60 328 347 323 323
query61 129 134 126 126
query62 752 751 662 662
query63 225 189 188 188
query64 1819 1037 672 672
query65 4319 4214 4220 4214
query66 728 404 306 306
query67 15801 15633 15653 15633
query68 8364 912 522 522
query69 534 299 268 268
query70 1204 1087 1110 1087
query71 518 321 298 298
query72 5826 4777 4892 4777
query73 1440 693 362 362
query74 9212 9132 8946 8946
query75 4062 3232 2710 2710
query76 4187 1295 760 760
query77 717 434 282 282
query78 10254 10126 9420 9420
query79 2313 834 578 578
query80 595 518 430 430
query81 478 262 220 220
query82 444 127 97 97
query83 258 255 240 240
query84 298 109 84 84
query85 789 358 314 314
query86 424 291 284 284
query87 4407 4354 4357 4354
query88 3724 2282 2296 2282
query89 411 313 277 277
query90 1820 208 206 206
query91 142 146 112 112
query92 76 60 57 57
query93 1845 941 580 580
query94 664 428 273 273
query95 370 294 277 277
query96 504 612 284 284
query97 3189 3229 3105 3105
query98 241 205 205 205
query99 1480 1385 1301 1301
Total cold run time: 300205 ms
Total hot run time: 193979 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](hive) fix write hive partition by Doris [doris]
Posted by "Thearas (via GitHub)" <gi...@apache.org>.
Thearas commented on PR #50864:
URL: https://github.com/apache/doris/pull/50864#issuecomment-2877038666
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](hive) fix write hive partition by Doris [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #50864:
URL: https://github.com/apache/doris/pull/50864#issuecomment-2877270350
<details>
<summary>ClickBench: <b>Total hot run time: 29.02 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit acaed5e681e8399e30f8cbbe396fb3a9072336c5, data reload: false
query1 0.04 0.04 0.03
query2 0.11 0.10 0.12
query3 0.24 0.19 0.19
query4 1.60 0.19 0.19
query5 0.61 0.58 0.61
query6 1.18 0.73 0.73
query7 0.03 0.02 0.02
query8 0.04 0.03 0.03
query9 0.57 0.52 0.52
query10 0.56 0.57 0.58
query11 0.17 0.10 0.11
query12 0.15 0.11 0.12
query13 0.61 0.59 0.60
query14 0.77 0.80 0.80
query15 0.86 0.87 0.83
query16 0.39 0.38 0.38
query17 1.05 1.05 1.06
query18 0.22 0.22 0.21
query19 1.91 1.78 1.78
query20 0.01 0.01 0.02
query21 15.40 0.94 0.56
query22 0.76 1.13 0.64
query23 14.99 1.38 0.63
query24 6.90 2.09 0.65
query25 0.49 0.21 0.05
query26 0.60 0.15 0.14
query27 0.05 0.06 0.04
query28 9.68 0.88 0.45
query29 12.62 3.97 3.30
query30 0.26 0.09 0.07
query31 2.81 0.59 0.39
query32 3.30 0.54 0.48
query33 3.08 3.03 3.00
query34 15.79 5.02 4.49
query35 4.49 4.53 4.49
query36 0.69 0.50 0.49
query37 0.08 0.06 0.06
query38 0.05 0.04 0.03
query39 0.03 0.02 0.02
query40 0.17 0.14 0.14
query41 0.08 0.03 0.03
query42 0.03 0.02 0.02
query43 0.04 0.04 0.03
Total cold run time: 103.51 s
Total hot run time: 29.02 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](hive) fix write hive partition by Doris [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #50864:
URL: https://github.com/apache/doris/pull/50864#issuecomment-2877213299
<details>
<summary>TPC-H: <b>Total hot run time: 33478 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit acaed5e681e8399e30f8cbbe396fb3a9072336c5, data reload: false
------ Round 1 ----------------------------------
q1 25894 4989 4938 4938
q2 2064 292 183 183
q3 10552 1252 678 678
q4 10540 1004 548 548
q5 7568 2355 2335 2335
q6 187 161 137 137
q7 915 747 611 611
q8 9337 1251 1075 1075
q9 6815 5047 5091 5047
q10 6857 2313 1863 1863
q11 496 280 265 265
q12 348 346 216 216
q13 17759 3684 3028 3028
q14 222 222 206 206
q15 512 484 481 481
q16 408 428 368 368
q17 592 844 352 352
q18 7509 7265 7040 7040
q19 1894 966 556 556
q20 332 324 214 214
q21 3889 2595 2369 2369
q22 1001 1018 968 968
Total cold run time: 115691 ms
Total hot run time: 33478 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5138 5031 5018 5018
q2 237 318 240 240
q3 2154 2618 2271 2271
q4 1340 1738 1343 1343
q5 4373 4326 4409 4326
q6 223 179 136 136
q7 1979 1902 1756 1756
q8 2563 2599 2532 2532
q9 7276 7225 6993 6993
q10 3063 3261 2764 2764
q11 587 505 487 487
q12 675 769 590 590
q13 3503 3915 3227 3227
q14 292 286 269 269
q15 530 493 478 478
q16 487 476 430 430
q17 1122 1522 1383 1383
q18 7683 7554 7533 7533
q19 836 816 818 816
q20 2036 1974 1819 1819
q21 5103 4813 4583 4583
q22 1106 1022 1035 1022
Total cold run time: 52306 ms
Total hot run time: 50016 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](hive) fix write hive partition by Doris [doris]
Posted by "suxiaogang223 (via GitHub)" <gi...@apache.org>.
suxiaogang223 commented on PR #50864:
URL: https://github.com/apache/doris/pull/50864#issuecomment-2877044606
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org