You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "Baymine (via GitHub)" <gi...@apache.org> on 2026/03/23 05:58:58 UTC
[PR] [fix](csv) Honor Hive CSV physical line boundaries [doris]
Baymine opened a new pull request, #61614:
URL: https://github.com/apache/doris/pull/61614
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: Hive CSV external tables that use TextInputFormat should keep one record per physical line. The current CSV reader can continue scanning across line delimiters when an enclosure remains open, which produces wrong query results for quoted values containing newlines.
### Release note
Fix wrong results when querying Hive CSV external tables with quoted newlines.
### Check List (For Author)
- Test: No need to test (will be run separately by the user)
- Behavior changed: Yes (Hive CSV external table queries now stop at physical line boundaries instead of merging lines through an open quote)
- Does this need documentation: No
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](csv) Honor Hive CSV physical line boundaries [doris]
Posted by "Baymine (via GitHub)" <gi...@apache.org>.
Baymine commented on PR #61614:
URL: https://github.com/apache/doris/pull/61614#issuecomment-4404479856
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](csv) Honor Hive CSV physical line boundaries [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #61614:
URL: https://github.com/apache/doris/pull/61614#discussion_r2975950191
##########
be/src/format/csv/csv_reader.cpp:
##########
@@ -585,6 +585,10 @@ Status CsvReader::_create_file_reader(bool need_schema) {
Status CsvReader::_create_line_reader() {
std::shared_ptr<TextLineReaderContextIf> text_line_reader_ctx;
+ // Hive TextInputFormat defines one record per physical line, so OpenCSVSerde must not
+ // merge rows across line delimiters even if a quote stays open.
+ const bool allow_multiline_records = !(_range.__isset.table_format_params &&
+ _range.table_format_params.table_format_type == "hive");
Review Comment:
[medium] Per AGENTS.md, behavioral changes should have regression tests. The PR states "No need to test (will be run separately by the user)" but this is a user-visible behavior change to Hive CSV external table queries. A regression test under `regression-test/` exercising a Hive external table with CSV data containing quoted newlines would provide end-to-end confidence and prevent future regressions.
Also consider: should this check be broadened beyond just `"hive"`? For example, if `table_format_type` is `"transactional_hive"`, should it also use single-line mode? The current code only checks for `"hive"` but `file_scanner.cpp` also handles `"transactional_hive"` as a distinct type.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](csv) Honor Hive CSV physical line boundaries [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #61614:
URL: https://github.com/apache/doris/pull/61614#issuecomment-4108430160
<details>
<summary>TPC-H: <b>Total hot run time: 26906 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d97ab687e0c523932bd3a35a26cca7f8fd88c48a, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17621 4551 4330 4330
q2 q3 10638 792 519 519
q4 4685 352 250 250
q5 7573 1210 1013 1013
q6 178 175 145 145
q7 773 848 679 679
q8 9314 1499 1337 1337
q9 5072 4775 4755 4755
q10 6301 1888 1688 1688
q11 483 258 245 245
q12 788 597 465 465
q13 18037 2936 2236 2236
q14 236 233 218 218
q15 q16 738 739 677 677
q17 766 854 434 434
q18 5904 5352 5155 5155
q19 1103 981 627 627
q20 550 493 377 377
q21 4516 1884 1492 1492
q22 528 388 264 264
Total cold run time: 95804 ms
Total hot run time: 26906 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4820 4606 4571 4571
q2 q3 3912 4342 3831 3831
q4 896 1248 801 801
q5 4139 4461 4300 4300
q6 187 178 149 149
q7 1780 1652 1537 1537
q8 2479 2700 2553 2553
q9 7613 7399 7453 7399
q10 3733 4105 3687 3687
q11 513 437 417 417
q12 502 584 461 461
q13 2771 3272 2311 2311
q14 302 306 289 289
q15 q16 729 776 740 740
q17 1168 1341 1370 1341
q18 7260 6793 7081 6793
q19 956 902 911 902
q20 2124 2183 2024 2024
q21 3928 3542 3289 3289
q22 493 454 408 408
Total cold run time: 50305 ms
Total hot run time: 47803 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](csv) Honor Hive CSV physical line boundaries [doris]
Posted by "Thearas (via GitHub)" <gi...@apache.org>.
Thearas commented on PR #61614:
URL: https://github.com/apache/doris/pull/61614#issuecomment-4108206039
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](csv) Honor Hive CSV physical line boundaries [doris]
Posted by "Baymine (via GitHub)" <gi...@apache.org>.
Baymine commented on PR #61614:
URL: https://github.com/apache/doris/pull/61614#issuecomment-4108304602
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](csv) Honor Hive CSV physical line boundaries [doris]
Posted by "zclllyybb (via GitHub)" <gi...@apache.org>.
zclllyybb commented on PR #61614:
URL: https://github.com/apache/doris/pull/61614#issuecomment-4111655958
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](csv) Honor Hive CSV physical line boundaries [doris]
Posted by "Baymine (via GitHub)" <gi...@apache.org>.
Baymine commented on PR #61614:
URL: https://github.com/apache/doris/pull/61614#issuecomment-4404478668
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](csv) Honor Hive CSV physical line boundaries [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #61614:
URL: https://github.com/apache/doris/pull/61614#issuecomment-4108472925
<details>
<summary>TPC-DS: <b>Total hot run time: 169047 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d97ab687e0c523932bd3a35a26cca7f8fd88c48a, data reload: false
query5 4330 635 526 526
query6 336 225 214 214
query7 4228 485 277 277
query8 351 241 229 229
query9 8717 2739 2731 2731
query10 540 409 338 338
query11 7016 5116 4893 4893
query12 183 136 127 127
query13 1280 457 344 344
query14 5742 3737 3494 3494
query14_1 2867 2880 2813 2813
query15 202 190 176 176
query16 966 475 446 446
query17 880 713 602 602
query18 2437 444 344 344
query19 213 199 182 182
query20 140 124 125 124
query21 210 134 113 113
query22 13332 14379 14921 14379
query23 16286 15946 15627 15627
query23_1 15862 15678 15555 15555
query24 7172 1622 1224 1224
query24_1 1227 1229 1220 1220
query25 551 466 403 403
query26 1239 263 150 150
query27 2756 483 310 310
query28 4469 1858 1836 1836
query29 795 584 479 479
query30 295 232 186 186
query31 1025 942 878 878
query32 78 68 70 68
query33 514 330 284 284
query34 882 880 525 525
query35 656 683 601 601
query36 1063 1152 1028 1028
query37 135 95 78 78
query38 3000 2903 2885 2885
query39 851 838 805 805
query39_1 800 801 817 801
query40 229 190 132 132
query41 62 59 60 59
query42 267 260 269 260
query43 240 253 221 221
query44
query45 194 191 182 182
query46 888 997 603 603
query47 2099 2132 2049 2049
query48 319 328 242 242
query49 629 483 383 383
query50 742 287 228 228
query51 4126 4048 4020 4020
query52 268 277 260 260
query53 287 340 287 287
query54 317 267 269 267
query55 90 89 82 82
query56 319 332 311 311
query57 1925 1770 1598 1598
query58 293 277 263 263
query59 2789 3009 2740 2740
query60 350 343 332 332
query61 158 154 154 154
query62 627 605 537 537
query63 315 283 274 274
query64 4960 1320 1104 1104
query65
query66 1483 479 406 406
query67 24245 24348 24175 24175
query68
query69 422 331 295 295
query70 955 1011 971 971
query71 349 315 303 303
query72 3041 2604 2625 2604
query73 549 565 323 323
query74 9654 9543 9400 9400
query75 2879 2776 2497 2497
query76 2283 1045 672 672
query77 366 393 301 301
query78 10944 11157 10481 10481
query79 2352 767 597 597
query80 1737 625 549 549
query81 546 271 221 221
query82 990 154 120 120
query83 343 263 242 242
query84 300 120 104 104
query85 914 498 460 460
query86 411 325 293 293
query87 3152 3139 2992 2992
query88 3628 2703 2683 2683
query89 425 376 346 346
query90 2027 177 170 170
query91 176 171 139 139
query92 77 72 74 72
query93 1054 849 504 504
query94 645 319 281 281
query95 598 334 324 324
query96 652 514 225 225
query97 2481 2476 2387 2387
query98 232 230 223 223
query99 1000 997 920 920
Total cold run time: 251108 ms
Total hot run time: 169047 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](csv) Honor Hive CSV physical line boundaries [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #61614:
URL: https://github.com/apache/doris/pull/61614#issuecomment-4108576179
# BE UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/d97ab687e0c523932bd3a35a26cca7f8fd88c48a_d97ab687e0c523932bd3a35a26cca7f8fd88c48a/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/d97ab687e0c523932bd3a35a26cca7f8fd88c48a_d97ab687e0c523932bd3a35a26cca7f8fd88c48a/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 52.83% (19863/37596) |
| Line Coverage | 36.30% (185379/510655) |
| Region Coverage | 32.57% (143646/441014) |
| Branch Coverage | 33.76% (62870/186233) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](csv) Honor Hive CSV physical line boundaries [doris]
Posted by "Baymine (via GitHub)" <gi...@apache.org>.
Baymine commented on code in PR #61614:
URL: https://github.com/apache/doris/pull/61614#discussion_r3206962511
##########
be/src/format/csv/csv_reader.cpp:
##########
@@ -585,6 +585,10 @@ Status CsvReader::_create_file_reader(bool need_schema) {
Status CsvReader::_create_line_reader() {
std::shared_ptr<TextLineReaderContextIf> text_line_reader_ctx;
+ // Hive TextInputFormat defines one record per physical line, so OpenCSVSerde must not
+ // merge rows across line delimiters even if a quote stays open.
+ const bool allow_multiline_records = !(_range.__isset.table_format_params &&
+ _range.table_format_params.table_format_type == "hive");
Review Comment:
This path isn't reachable for the CSV reader. table_format_type == "transactional_hive" is only set by `HiveScanNode.java:468` when HiveSplit.isACID() is true, and Hive ACID requires ORC storage by spec — ACID tables cannot be stored as TEXTFILE/CSV, so csv_reader.cpp::_create_line_reader() is never entered on that path. The parallel `new_json_reader.cpp:388` follows the same "hive"-only convention. I'd prefer to keep the check narrow rather than add code for an unreachable branch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org