You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "HHoflittlefish777 (via GitHub)" <gi...@apache.org> on 2023/12/01 11:55:59 UTC
[PR] [improve](csv_reader) handle csv reader error [doris]
HHoflittlefish777 opened a new pull request, #27892:
URL: https://github.com/apache/doris/pull/27892
## Proposed changes
Handle csv reader error.
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](csv_reader) handle csv reader error [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27892:
URL: https://github.com/apache/doris/pull/27892#issuecomment-1838448154
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit 693c60b7b95fc7fa6e63222b3fa2a047701a3f94, data reload: true
run tpch-sf100 query with default conf and session variables
q1 4945 4602 4607 4602
q2 398 126 117 117
q3 1535 1331 1270 1270
q4 1174 945 936 936
q5 3244 3247 3234 3234
q6 244 122 121 121
q7 982 492 496 492
q8 2347 2214 2190 2190
q9 7036 6944 6940 6940
q10 3294 3387 3396 3387
q11 340 204 208 204
q12 353 219 208 208
q13 4672 3911 3913 3911
q14 249 216 201 201
q15 582 530 527 527
q16 430 395 400 395
q17 1033 592 561 561
q18 7869 9419 7548 7548
q19 1573 1559 1533 1533
q20 576 314 335 314
q21 3423 2956 2942 2942
q22 364 298 305 298
Total cold run time: 46663 ms
Total hot run time: 41931 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4625 4585 4574 4574
q2 315 202 213 202
q3 3768 3741 3756 3741
q4 2538 2534 2526 2526
q5 6167 6191 6188 6188
q6 240 118 124 118
q7 2712 2047 2018 2018
q8 3719 3711 3731 3711
q9 9502 9393 9425 9393
q10 4077 4147 4146 4146
q11 644 510 519 510
q12 804 621 619 619
q13 4387 3669 3635 3635
q14 272 243 239 239
q15 574 533 520 520
q16 521 489 490 489
q17 2104 2057 2088 2057
q18 9673 9098 9098 9098
q19 1817 1753 1764 1753
q20 2318 1981 1975 1975
q21 7232 7026 6994 6994
q22 658 579 578 578
Total cold run time: 68667 ms
Total hot run time: 65084 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](csv_reader) handle csv reader error [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27892:
URL: https://github.com/apache/doris/pull/27892#issuecomment-1836045952
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.1 seconds
stream load tsv: 565 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 34 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 30.0 seconds inserted 10000000 Rows, about 333K ops/s
storage size: 17164083340 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](csv_reader) handle csv reader error [doris]
Posted by "HHoflittlefish777 (via GitHub)" <gi...@apache.org>.
HHoflittlefish777 commented on PR #27892:
URL: https://github.com/apache/doris/pull/27892#issuecomment-1835999634
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](csv_reader) handle csv reader error [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring merged PR #27892:
URL: https://github.com/apache/doris/pull/27892
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](csv_reader) handle csv reader error [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27892:
URL: https://github.com/apache/doris/pull/27892#issuecomment-1836041842
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](csv_reader) handle csv reader error [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #27892:
URL: https://github.com/apache/doris/pull/27892#discussion_r1412022940
##########
be/src/vec/exec/scan/vfile_scanner.cpp:
##########
@@ -713,7 +713,7 @@ void VFileScanner::_truncate_char_or_varchar_column(Block* block, int idx, int l
Status VFileScanner::_get_next_reader() {
Review Comment:
warning: function '_get_next_reader' has cognitive complexity of 106 (threshold 50) [readability-function-cognitive-complexity]
```cpp
Status VFileScanner::_get_next_reader() {
^
```
<details>
<summary>Additional context</summary>
**be/src/vec/exec/scan/vfile_scanner.cpp:713:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
while (true) {
^
```
**be/src/vec/exec/scan/vfile_scanner.cpp:714:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (_cur_reader) {
^
```
**be/src/vec/exec/scan/vfile_scanner.cpp:715:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(_cur_reader->close());
^
```
**be/src/common/status.h:523:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/vec/exec/scan/vfile_scanner.cpp:715:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
RETURN_IF_ERROR(_cur_reader->close());
^
```
**be/src/common/status.h:525:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/vec/exec/scan/vfile_scanner.cpp:719:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (_next_range >= _ranges.size()) {
^
```
**be/src/vec/exec/scan/vfile_scanner.cpp:724:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (_next_range != 0) {
^
```
**be/src/vec/exec/scan/vfile_scanner.cpp:737:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (format_type == TFileFormatType::FORMAT_JNI && range.__isset.table_format_params &&
^
```
**be/src/vec/exec/scan/vfile_scanner.cpp:739:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (range.table_format_params.hudi_params.delta_logs.empty()) {
^
```
**be/src/vec/exec/scan/vfile_scanner.cpp:745:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
switch (format_type) {
^
```
**be/src/vec/exec/scan/vfile_scanner.cpp:747:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (_real_tuple_desc->table_desc()->table_type() ==
^
```
**be/src/vec/exec/scan/vfile_scanner.cpp:756:** +1, nesting level increased to 3
```cpp
} else if (range.__isset.table_format_params &&
^
```
**be/src/vec/exec/scan/vfile_scanner.cpp:756:** +1
```cpp
} else if (range.__isset.table_format_params &&
^
```
**be/src/vec/exec/scan/vfile_scanner.cpp:762:** +1, nesting level increased to 3
```cpp
} else if (range.__isset.table_format_params &&
^
```
**be/src/vec/exec/scan/vfile_scanner.cpp:762:** +1
```cpp
} else if (range.__isset.table_format_params &&
^
```
**be/src/vec/exec/scan/vfile_scanner.cpp:782:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(parquet_reader->open());
^
```
**be/src/common/status.h:523:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/vec/exec/scan/vfile_scanner.cpp:782:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
RETURN_IF_ERROR(parquet_reader->open());
^
```
**be/src/common/status.h:525:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/vec/exec/scan/vfile_scanner.cpp:784:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (push_down_predicates) {
^
```
**be/src/vec/exec/scan/vfile_scanner.cpp:785:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
RETURN_IF_ERROR(_process_late_arrival_conjuncts());
^
```
**be/src/common/status.h:523:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/vec/exec/scan/vfile_scanner.cpp:785:** +5, including nesting penalty of 4, nesting level increased to 5
```cpp
RETURN_IF_ERROR(_process_late_arrival_conjuncts());
^
```
**be/src/common/status.h:525:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/vec/exec/scan/vfile_scanner.cpp:787:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (range.__isset.table_format_params &&
^
```
**be/src/vec/exec/scan/vfile_scanner.cpp:787:** +1
```cpp
if (range.__isset.table_format_params &&
^
```
**be/src/vec/exec/scan/vfile_scanner.cpp:798:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
RETURN_IF_ERROR(iceberg_reader->init_row_filters(range));
^
```
**be/src/common/status.h:523:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/vec/exec/scan/vfile_scanner.cpp:798:** +5, including nesting penalty of 4, nesting level increased to 5
```cpp
RETURN_IF_ERROR(iceberg_reader->init_row_filters(range));
^
```
**be/src/common/status.h:525:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/vec/exec/scan/vfile_scanner.cpp:800:** +1, nesting level increased to 3
```cpp
} else {
^
```
**be/src/vec/exec/scan/vfile_scanner.cpp:816:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (push_down_predicates) {
^
```
**be/src/vec/exec/scan/vfile_scanner.cpp:817:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
RETURN_IF_ERROR(_process_late_arrival_conjuncts());
^
```
**be/src/common/status.h:523:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/vec/exec/scan/vfile_scanner.cpp:817:** +5, including nesting penalty of 4, nesting level increased to 5
```cpp
RETURN_IF_ERROR(_process_late_arrival_conjuncts());
^
```
**be/src/common/status.h:525:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/vec/exec/scan/vfile_scanner.cpp:819:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (range.__isset.table_format_params &&
^
```
**be/src/vec/exec/scan/vfile_scanner.cpp:819:** +1
```cpp
if (range.__isset.table_format_params &&
^
```
**be/src/vec/exec/scan/vfile_scanner.cpp:829:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
RETURN_IF_ERROR(tran_orc_reader->init_row_filters(range));
^
```
**be/src/common/status.h:523:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/vec/exec/scan/vfile_scanner.cpp:829:** +5, including nesting penalty of 4, nesting level increased to 5
```cpp
RETURN_IF_ERROR(tran_orc_reader->init_row_filters(range));
^
```
**be/src/common/status.h:525:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/vec/exec/scan/vfile_scanner.cpp:831:** +1, nesting level increased to 3
```cpp
} else {
^
```
**be/src/vec/exec/scan/vfile_scanner.cpp:879:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (init_status.is<END_OF_FILE>()) {
^
```
**be/src/vec/exec/scan/vfile_scanner.cpp:882:** +1, nesting level increased to 2
```cpp
} else if (!init_status.ok()) {
^
```
**be/src/vec/exec/scan/vfile_scanner.cpp:883:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (init_status.is<ErrorCode::NOT_FOUND>()) {
^
```
**be/src/vec/exec/scan/vfile_scanner.cpp:897:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(_generate_fill_columns());
^
```
**be/src/common/status.h:523:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/vec/exec/scan/vfile_scanner.cpp:897:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(_generate_fill_columns());
^
```
**be/src/common/status.h:525:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/vec/exec/scan/vfile_scanner.cpp:898:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (VLOG_NOTICE_IS_ON && !_missing_cols.empty() && _is_load) {
^
```
**be/src/vec/exec/scan/vfile_scanner.cpp:910:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (_state->query_options().truncate_char_or_varchar_columns && need_to_get_parsed_schema) {
^
```
**be/src/vec/exec/scan/vfile_scanner.cpp:910:** +1
```cpp
if (_state->query_options().truncate_char_or_varchar_columns && need_to_get_parsed_schema) {
^
```
**be/src/vec/exec/scan/vfile_scanner.cpp:913:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (!status.ok() && status.code() != TStatusCode::NOT_IMPLEMENTED_ERROR) {
^
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](csv_reader) handle csv reader error [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27892:
URL: https://github.com/apache/doris/pull/27892#issuecomment-1836041893
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org