You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "amorynan (via GitHub)" <gi...@apache.org> on 2023/11/14 10:36:47 UTC
[PR] [FIX](complextype)fix struct nested complex collection type and and regresstest [doris]
amorynan opened a new pull request, #26973:
URL: https://github.com/apache/doris/pull/26973
## Proposed changes
this pr do
1. fix array/map/struct write next_rowid for num_rows when we read from segmentfile
2.improve struct can nested complex type for load action
3.add one/two-level regress test for complex types
Issue Number: close #xxx
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [FIX](complextype)fix struct nested complex collection type and and regresstest [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26973:
URL: https://github.com/apache/doris/pull/26973#issuecomment-1818183580
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [FIX](complextype)fix struct nested complex collection type and and regresstest [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26973:
URL: https://github.com/apache/doris/pull/26973#issuecomment-1818236409
TeamCity be ut coverage result:
Function Coverage: 36.51% (8450/23143)
Line Coverage: 28.87% (68687/237935)
Region Coverage: 27.83% (35521/127623)
Branch Coverage: 24.58% (18111/73670)
Coverage Report: http://coverage.selectdb-in.cc/coverage/16a3ed8660cb5b8fcd42b337d273da7f554d721c_16a3ed8660cb5b8fcd42b337d273da7f554d721c/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [FIX](complextype)fix struct nested complex collection type and and regresstest [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26973:
URL: https://github.com/apache/doris/pull/26973#issuecomment-1818396075
TeamCity be ut coverage result:
Function Coverage: 36.53% (8450/23134)
Line Coverage: 28.88% (68692/237888)
Region Coverage: 27.84% (35524/127607)
Branch Coverage: 24.59% (18113/73662)
Coverage Report: http://coverage.selectdb-in.cc/coverage/4dd83dd39e20ac1527c03a996abe0009b2a8d391_4dd83dd39e20ac1527c03a996abe0009b2a8d391/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [FIX](complextype)fix struct nested complex collection type and and regresstest [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26973:
URL: https://github.com/apache/doris/pull/26973#issuecomment-1814017176
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit 18caea5bad584f6fff71d1da6ef0e045e54aa311, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4997 4654 4605 4605
q2 351 137 137 137
q3 2034 1858 1839 1839
q4 1392 1291 1267 1267
q5 4006 4005 4061 4005
q6 241 130 135 130
q7 1406 884 899 884
q8 2765 2807 2786 2786
q9 9685 9585 9396 9396
q10 3498 3583 3576 3576
q11 380 256 249 249
q12 446 296 298 296
q13 4583 3796 3824 3796
q14 321 290 286 286
q15 579 526 519 519
q16 674 585 588 585
q17 1141 984 982 982
q18 7816 7471 7362 7362
q19 1678 1703 1694 1694
q20 535 328 290 290
q21 4437 3990 3978 3978
q22 486 378 381 378
Total cold run time: 53451 ms
Total hot run time: 49040 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4615 4580 4588 4580
q2 341 230 265 230
q3 4031 4024 4022 4022
q4 2718 2703 2708 2703
q5 9709 9725 9678 9678
q6 241 127 126 126
q7 3030 2522 2482 2482
q8 4393 4475 4431 4431
q9 12933 12907 12849 12849
q10 4134 4184 4203 4184
q11 812 673 647 647
q12 990 816 795 795
q13 4322 3545 3565 3545
q14 378 342 346 342
q15 575 529 517 517
q16 774 658 691 658
q17 3815 3885 3901 3885
q18 9643 9056 8896 8896
q19 1836 1828 1786 1786
q20 2422 2042 2034 2034
q21 8783 8544 8439 8439
q22 949 851 846 846
Total cold run time: 81444 ms
Total hot run time: 77675 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [FIX](complextype)fix struct nested complex collection type and and regresstest [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26973:
URL: https://github.com/apache/doris/pull/26973#issuecomment-1811986876
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [FIX](complextype)fix struct nested complex collection type and and regresstest [doris]
Posted by "amorynan (via GitHub)" <gi...@apache.org>.
amorynan commented on PR #26973:
URL: https://github.com/apache/doris/pull/26973#issuecomment-1818407448
run pipelinex_p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [FIX](complextype)fix struct nested complex collection type and and regresstest [doris]
Posted by "amorynan (via GitHub)" <gi...@apache.org>.
amorynan commented on PR #26973:
URL: https://github.com/apache/doris/pull/26973#issuecomment-1818358547
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [FIX](complextype)fix struct nested complex collection type and and regresstest [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26973:
URL: https://github.com/apache/doris/pull/26973#issuecomment-1813865446
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [FIX](complextype)fix struct nested complex collection type and and regresstest [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26973:
URL: https://github.com/apache/doris/pull/26973#issuecomment-1813936754
TeamCity be ut coverage result:
Function Coverage: 36.78% (8410/22868)
Line Coverage: 29.29% (68415/233563)
Region Coverage: 27.86% (35345/126849)
Branch Coverage: 24.61% (18059/73374)
Coverage Report: http://coverage.selectdb-in.cc/coverage/18caea5bad584f6fff71d1da6ef0e045e54aa311_18caea5bad584f6fff71d1da6ef0e045e54aa311/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [FIX](complextype)fix struct nested complex collection type and and regresstest [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26973:
URL: https://github.com/apache/doris/pull/26973#issuecomment-1818617550
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [FIX](complextype)fix struct nested complex collection type and and regresstest [doris]
Posted by "amorynan (via GitHub)" <gi...@apache.org>.
amorynan commented on PR #26973:
URL: https://github.com/apache/doris/pull/26973#issuecomment-1815642044
run pipelinex_p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [FIX](complextype)fix struct nested complex collection type and and regresstest [doris]
Posted by "xiaokang (via GitHub)" <gi...@apache.org>.
xiaokang merged PR #26973:
URL: https://github.com/apache/doris/pull/26973
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [FIX](complextype)fix struct nested complex collection type and and regresstest [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26973:
URL: https://github.com/apache/doris/pull/26973#issuecomment-1818203279
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit 16a3ed8660cb5b8fcd42b337d273da7f554d721c, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4932 4683 4687 4683
q2 356 161 159 159
q3 2034 1950 1847 1847
q4 1379 1289 1223 1223
q5 3934 3959 4015 3959
q6 252 126 130 126
q7 1392 878 894 878
q8 2780 2771 2760 2760
q9 9985 9451 9572 9451
q10 3467 3512 3523 3512
q11 381 250 249 249
q12 440 296 293 293
q13 4564 3819 3808 3808
q14 316 286 297 286
q15 584 526 522 522
q16 674 578 582 578
q17 1126 975 942 942
q18 7915 7560 7527 7527
q19 1660 1670 1657 1657
q20 571 294 289 289
q21 4413 3933 4017 3933
q22 475 367 374 367
Total cold run time: 53630 ms
Total hot run time: 49049 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4609 4558 4577 4558
q2 348 230 263 230
q3 4010 4000 3999 3999
q4 2719 2711 2711 2711
q5 9602 9576 9616 9576
q6 252 123 124 123
q7 3021 2464 2463 2463
q8 4451 4454 4447 4447
q9 12892 12851 12866 12851
q10 4054 4161 4158 4158
q11 795 664 655 655
q12 976 833 818 818
q13 4273 3595 3559 3559
q14 379 345 342 342
q15 571 517 523 517
q16 736 692 662 662
q17 3914 3894 3934 3894
q18 9623 9217 9097 9097
q19 1804 1767 1781 1767
q20 2416 2089 2048 2048
q21 8839 8845 8724 8724
q22 878 802 788 788
Total cold run time: 81162 ms
Total hot run time: 77987 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [FIX](complextype)fix struct nested complex collection type and and regresstest [doris]
Posted by "amorynan (via GitHub)" <gi...@apache.org>.
amorynan commented on PR #26973:
URL: https://github.com/apache/doris/pull/26973#issuecomment-1813860209
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [FIX](complextype)fix struct nested complex collection type and and regresstest [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26973:
URL: https://github.com/apache/doris/pull/26973#issuecomment-1809960360
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [FIX](complextype)fix struct nested complex collection type and and regresstest [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26973:
URL: https://github.com/apache/doris/pull/26973#issuecomment-1818207196
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.67 seconds
stream load tsv: 566 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.6 seconds inserted 10000000 Rows, about 349K ops/s
storage size: 17100218325 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [FIX](complextype)fix struct nested complex collection type and and regresstest [doris]
Posted by "amorynan (via GitHub)" <gi...@apache.org>.
amorynan commented on PR #26973:
URL: https://github.com/apache/doris/pull/26973#issuecomment-1814423237
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [FIX](complextype)fix struct nested complex collection type and and regresstest [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26973:
URL: https://github.com/apache/doris/pull/26973#issuecomment-1814673079
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.45 seconds
stream load tsv: 566 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17098963746 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [FIX](complextype)fix struct nested complex collection type and and regresstest [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26973:
URL: https://github.com/apache/doris/pull/26973#issuecomment-1813865402
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [FIX](complextype)fix struct nested complex collection type and and regresstest [doris]
Posted by "amorynan (via GitHub)" <gi...@apache.org>.
amorynan commented on code in PR #26973:
URL: https://github.com/apache/doris/pull/26973#discussion_r1398630717
##########
be/src/vec/data_types/serde/data_type_struct_serde.cpp:
##########
@@ -94,106 +93,125 @@ Status DataTypeStructSerDe::deserialize_one_cell_from_json(IColumn& column, Slic
}
return Status::OK();
}
-
- ReadBuffer rb(slice.data, slice.size);
- ++rb.position();
+ // remove '{' '}'
+ slice.remove_prefix(1);
+ slice.remove_suffix(1);
+ slice.trim_prefix();
bool is_explicit_names = false;
- std::vector<std::string> field_names;
- std::vector<ReadBuffer> field_rbs;
- std::vector<size_t> field_pos;
-
- while (!rb.eof()) {
- StringRef slot(rb.position(), rb.count());
- bool has_quota = false;
- bool is_name = false;
- if (!next_slot_from_string(rb, slot, is_name, has_quota)) {
- return Status::InvalidArgument("Cannot read struct field from text '{}'",
- slot.to_string());
- }
- if (is_name) {
- std::string name = slot.to_string();
- if (!next_slot_from_string(rb, slot, is_name, has_quota)) {
- return Status::InvalidArgument("Cannot read struct field from text '{}'",
- slot.to_string());
- }
- ReadBuffer field_rb(const_cast<char*>(slot.data), slot.size);
- field_names.push_back(name);
- field_rbs.push_back(field_rb);
-
- if (!is_explicit_names) {
- is_explicit_names = true;
+ int nested_level = 0;
+ bool has_quote = false;
+ int start_pos = 0;
+ size_t slice_size = slice.size;
+ bool key_added = false;
+ int idx = 0;
+ char quote_char = 0;
+
+ auto elem_size = elemSerDeSPtrs.size();
+ int field_pos = 0;
+
+ for (; idx < slice_size; ++idx) {
+ char c = slice[idx];
+ if (c == '"' || c == '\'') {
+ if (!has_quote) {
+ quote_char = c;
+ has_quote = !has_quote;
+ } else if (has_quote && quote_char == c) {
+ quote_char = 0;
+ has_quote = !has_quote;
}
- } else {
- ReadBuffer field_rb(const_cast<char*>(slot.data), slot.size);
- field_rbs.push_back(field_rb);
- }
- }
-
- // TODO: should we support insert default field value when actual field number is less than
- // schema field number?
- if (field_rbs.size() != elemSerDeSPtrs.size()) {
- std::string cmp_str = field_rbs.size() > elemSerDeSPtrs.size() ? "more" : "less";
- return Status::InvalidArgument(
- "Actual struct field number {} is {} than schema field number {}.",
- field_rbs.size(), cmp_str, elemSerDeSPtrs.size());
- }
-
- if (is_explicit_names) {
- if (field_names.size() != field_rbs.size()) {
- return Status::InvalidArgument(
- "Struct field name number {} is not equal to field number {}.",
- field_names.size(), field_rbs.size());
- }
- std::unordered_set<std::string> name_set;
- for (size_t i = 0; i < field_names.size(); i++) {
- // check duplicate fields
- auto ret = name_set.insert(field_names[i]);
- if (!ret.second) {
- return Status::InvalidArgument("Struct field name {} is duplicate with others.",
- field_names[i]);
+ } else if (c == '\\' && idx + 1 < slice_size) { //escaped
+ ++idx;
+ } else if (!has_quote && (c == '[' || c == '{')) {
+ ++nested_level;
+ } else if (!has_quote && (c == ']' || c == '}')) {
+ --nested_level;
+ } else if (!has_quote && nested_level == 0 && c == options.map_key_delim && !key_added) {
Review Comment:
","
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [FIX](complextype)fix struct nested complex collection type and and regresstest [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26973:
URL: https://github.com/apache/doris/pull/26973#issuecomment-1814432148
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [FIX](complextype)fix struct nested complex collection type and and regresstest [doris]
Posted by "amorynan (via GitHub)" <gi...@apache.org>.
amorynan commented on PR #26973:
URL: https://github.com/apache/doris/pull/26973#issuecomment-1814115932
run p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [FIX](complextype)fix struct nested complex collection type and and regresstest [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26973:
URL: https://github.com/apache/doris/pull/26973#issuecomment-1814665906
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit 6acef23b0e81d666c381128fbff51b4d85ebcc62, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4940 4719 4675 4675
q2 354 156 157 156
q3 2044 1937 1914 1914
q4 1385 1279 1244 1244
q5 3963 3972 4082 3972
q6 251 132 133 132
q7 1417 866 894 866
q8 2755 2805 2801 2801
q9 9860 9788 9577 9577
q10 3465 3533 3530 3530
q11 380 249 239 239
q12 444 307 297 297
q13 4555 3846 3796 3796
q14 334 281 296 281
q15 587 525 526 525
q16 668 589 590 589
q17 1127 945 923 923
q18 7818 7335 7429 7335
q19 1695 1690 1658 1658
q20 579 325 295 295
q21 4443 3951 3980 3951
q22 487 383 381 381
Total cold run time: 53551 ms
Total hot run time: 49137 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4615 4547 4667 4547
q2 345 228 262 228
q3 4016 4024 3997 3997
q4 2718 2697 2702 2697
q5 9573 9590 9518 9518
q6 253 125 125 125
q7 2593 2292 2296 2292
q8 4438 4397 4496 4397
q9 13208 13173 13154 13154
q10 4090 4190 4174 4174
q11 800 682 704 682
q12 974 819 810 810
q13 4280 3578 3549 3549
q14 384 343 356 343
q15 581 518 537 518
q16 738 688 666 666
q17 3864 3917 3873 3873
q18 9533 9124 9080 9080
q19 1798 1804 1785 1785
q20 2405 2075 2075 2075
q21 8917 8456 8687 8456
q22 910 813 813 813
Total cold run time: 81033 ms
Total hot run time: 77779 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [FIX](complextype)fix struct nested complex collection type and and regresstest [doris]
Posted by "xiaokang (via GitHub)" <gi...@apache.org>.
xiaokang commented on code in PR #26973:
URL: https://github.com/apache/doris/pull/26973#discussion_r1396690419
##########
regression-test/suites/datatype_p0/nested_types/base_cases/one_level_nestedtypes_with_s3data.groovy:
##########
@@ -0,0 +1,283 @@
+import org.apache.commons.lang3.StringUtils
+
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("one_level_nestedtypes_with_s3data") {
+ sql """set enable_nereids_planner=false"""
+ sql """ set enable_fallback_to_original_planner=true;"""
+ // this test case aim to test one-level nested type with s3 data
+
+
+ String ak = getS3AK()
+ String sk = getS3SK()
+ String s3_endpoint = getS3Endpoint()
+ String bucket = context.config.otherConfigs.get("s3BucketName");
+
+
+ def dataFilePath = "https://"+"${bucket}"+"."+"${s3_endpoint}"+"/regression/datalake"
+// def dataFilePath = "/mnt/disk1/wangqiannan/export/ol"
+ def table_names = ["test_array_one_level", "test_map_one_level", "test_struct_one_level"]
+
+ def colNameArr = ["c_bool", "c_tinyint", "c_smallint", "c_int", "c_bigint", "c_largeint", "c_float",
+ "c_double", "c_decimal", "c_decimalv3", "c_date", "c_datetime", "c_datev2", "c_datetimev2",
+ "c_char", "c_varchar", "c_string"]
+
+ def groupby_or_orderby_exception = {is_groupby, table_name, col_name ->
+ test {
+ if (is_groupby) {
+ sql "select ${col_name} from ${table_name} group by ${col_name};"
+ } else {
+ sql "select ${col_name} from ${table_name} order by ${col_name};"
+ }
+ exception("errCode = 2, detailMessage = Doris hll, bitmap, array, map, struct, jsonb, variant column must use with specific function, and don't support filter, group by or order by")
+ }
+ }
+
+ def groupby_or_orderby_element_at = {is_groupby, table_name, agg_expr ->
+ if (is_groupby) {
+ order_qt_sql "select ${agg_expr} from ${table_name} where k1 IS NOT NULL group by ${agg_expr};"
+ } else {
+ order_qt_sql "select ${agg_expr} from ${table_name} where k1 IS NOT NULL order by ${agg_expr} limit 10;"
+ }
+ }
+
+ def be_id = 10139
+ def load_from_tvf = {table_name, uri_file, format ->
+ if (format == "csv") {
+ order_qt_sql_tvf """select * from local(
+ "file_path" = "${uri_file}",
+ "backend_id" = "${be_id}",
+ "column_separator"="|",
+ "format" = "${format}") order by c1 limit 10; """
+
+ sql """
+ insert into ${table_name} select * from local(
+ "file_path" = "${uri_file}",
+ "backend_id" = "${be_id}",
+ "column_separator"="|",
+ "format" = "${format}"); """
+ } else {
+ order_qt_sql_tvf """select * from local(
+ "file_path" = "${uri_file}",
+ "backend_id" = "${be_id}",
+ "column_separator"="|",
+ "format" = "${format}") order by k1 limit 10; """
+
+ sql """
+ insert into ${table_name} select * from local(
+ "file_path" = "${uri_file}",
+ "backend_id" = "${be_id}",
+ "column_separator"="|",
+ "format" = "${format}"); """
+ }
+ // where to filter different format data
+ qt_select_doris """ select * from ${table_name} where k1 IS NOT NULL order by k1 limit 10; """
+ }
+ def load_from_s3 = {table_name, uri_file, format ->
+ if (format == "csv") {
+ order_qt_sql_s3 """select * from s3(
+ "uri" = "${uri_file}",
+ "s3.access_key"= "${ak}",
+ "s3.secret_key" = "${sk}",
+ "format" = "${format}",
+ "column_separator"="|",
+ "read_json_by_line"="true") order by c1 limit 10; """
+
+ sql """
+ insert into ${table_name} select * from s3(
+ "uri" = "${uri_file}",
+ "s3.access_key"= "${ak}",
+ "s3.secret_key" = "${sk}",
+ "format" = "${format}",
+ "column_separator"="|",
+ "read_json_by_line"="true"); """
+ } else {
+ order_qt_sql_s3 """select * from s3(
+ "uri" = "${uri_file}",
+ "s3.access_key"= "${ak}",
+ "s3.secret_key" = "${sk}",
+ "format" = "${format}",
+ "read_json_by_line"="true") order by k1 limit 10; """
+
+ sql """
+ insert into ${table_name} select * from s3(
+ "uri" = "${uri_file}",
+ "s3.access_key"= "${ak}",
+ "s3.secret_key" = "${sk}",
+ "format" = "${format}",
+ "read_json_by_line"="true"); """
+ }
+ // where to filter different format data
+ qt_select_doris """ select * from ${table_name} where k1 IS NOT NULL order by k1 limit 10; """
+ }
+
+ // step1. create table
+ // step2. load from s3
+ // step 2.1 format: parquet|orc|json|csv
+ // step3. select *
+ // step4. select element_at(column in first, -1(last), null, 0)
+ // step5. select * from table where element_at(column) equals expr just ok
+ // step6. select * from table where groupby|orderby column
+ // step7. select * from table where groupby|orderby element_at(column)
+
+ def format_order = [
+ "parquet", "orc",
+// "json",
+ "csv"]
+ // create tables
+ for (int i = 0; i < table_names.size(); ++i) {
+ sql """ DROP TABLE IF EXISTS ${table_names[i]} """
+ String result = create_table_with_nested_type(1, [i], table_names[i])
+ sql result
+ }
+
+ //========================= ARRAY =========================
+ // insert into doris table
+ ArrayList<String> array_files = [
+ "${dataFilePath}/one_level_array.parquet",
+ "${dataFilePath}/one_level_array.orc",
+// "${dataFilePath}/one_level_array.json",
+ "${dataFilePath}/one_level_array.csv"
+ ]
+ int fi = 0
+ for (String f : array_files) {
+ sql "truncate table ${table_names[2]};"
+// load_from_tvf(table_names[0], f, format_order[fi])
+ load_from_s3(table_names[0], f, format_order[fi])
+ ++ fi
+ }
+ // select element_at(column)
+ for (String col : colNameArr) {
+ // first
+ order_qt_select_arr "select ${col}[1] from ${table_names[0]} where k1 IS NOT NULL order by k1 limit 10;"
+ // last
+ order_qt_select_arr "select ${col}[-1] from ${table_names[0]} where k1 IS NOT NULL order by k1 limit 10;"
+ // null
+ order_qt_select_arr_null "select ${col}[0] from ${table_names[0]} where k1 IS NOT NULL order by k1 limit 10;"
+ // null
+ order_qt_select_arr_null "select ${col}[1000] from ${table_names[0]} where k1 IS NOT NULL order by k1 limit 10;"
+ }
+ // select * from table where element_at(column) with equal expr
+ for (String col : colNameArr) {
+ order_qt_select_arr "select ${col}[1], ${col}[-1] from ${table_names[0]} where k1 IS NOT NULL AND ${col}[1]<${col}[-1] order by k1 limit 10;"
+ }
+ // select * from table where groupby|orderby column will meet exception
+ for (String col : colNameArr) {
+ groupby_or_orderby_exception(true, table_names[0], col)
+ groupby_or_orderby_exception(false, table_names[0], col)
+ }
+ // select * from table where groupby|orderby element_at(column)
+ for (String col : colNameArr) {
+ String agg_expr = "${col}[1]"
+ groupby_or_orderby_element_at(true, table_names[0], agg_expr)
+ groupby_or_orderby_element_at(false, table_names[0], agg_expr)
+ }
+
+ //========================== MAP ==========================
+ // insert into doris table
+ ArrayList<String> map_files = [
+ "${dataFilePath}/one_level_map.parquet",
+ "${dataFilePath}/one_level_map.orc",
+// "${dataFilePath}/one_level_map.json",
+ "${dataFilePath}/one_level_map.csv"
+ ]
+ fi = 0
+ for (String f : map_files) {
+ sql "truncate table ${table_names[2]};"
Review Comment:
table_names[1]
##########
regression-test/suites/datatype_p0/nested_types/base_cases/one_level_nestedtypes_with_s3data.groovy:
##########
@@ -0,0 +1,283 @@
+import org.apache.commons.lang3.StringUtils
+
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("one_level_nestedtypes_with_s3data") {
+ sql """set enable_nereids_planner=false"""
+ sql """ set enable_fallback_to_original_planner=true;"""
+ // this test case aim to test one-level nested type with s3 data
+
+
+ String ak = getS3AK()
+ String sk = getS3SK()
+ String s3_endpoint = getS3Endpoint()
+ String bucket = context.config.otherConfigs.get("s3BucketName");
+
+
+ def dataFilePath = "https://"+"${bucket}"+"."+"${s3_endpoint}"+"/regression/datalake"
+// def dataFilePath = "/mnt/disk1/wangqiannan/export/ol"
+ def table_names = ["test_array_one_level", "test_map_one_level", "test_struct_one_level"]
+
+ def colNameArr = ["c_bool", "c_tinyint", "c_smallint", "c_int", "c_bigint", "c_largeint", "c_float",
+ "c_double", "c_decimal", "c_decimalv3", "c_date", "c_datetime", "c_datev2", "c_datetimev2",
+ "c_char", "c_varchar", "c_string"]
+
+ def groupby_or_orderby_exception = {is_groupby, table_name, col_name ->
+ test {
+ if (is_groupby) {
+ sql "select ${col_name} from ${table_name} group by ${col_name};"
+ } else {
+ sql "select ${col_name} from ${table_name} order by ${col_name};"
+ }
+ exception("errCode = 2, detailMessage = Doris hll, bitmap, array, map, struct, jsonb, variant column must use with specific function, and don't support filter, group by or order by")
+ }
+ }
+
+ def groupby_or_orderby_element_at = {is_groupby, table_name, agg_expr ->
+ if (is_groupby) {
+ order_qt_sql "select ${agg_expr} from ${table_name} where k1 IS NOT NULL group by ${agg_expr};"
+ } else {
+ order_qt_sql "select ${agg_expr} from ${table_name} where k1 IS NOT NULL order by ${agg_expr} limit 10;"
+ }
+ }
+
+ def be_id = 10139
+ def load_from_tvf = {table_name, uri_file, format ->
+ if (format == "csv") {
+ order_qt_sql_tvf """select * from local(
+ "file_path" = "${uri_file}",
+ "backend_id" = "${be_id}",
+ "column_separator"="|",
+ "format" = "${format}") order by c1 limit 10; """
+
+ sql """
+ insert into ${table_name} select * from local(
+ "file_path" = "${uri_file}",
+ "backend_id" = "${be_id}",
+ "column_separator"="|",
+ "format" = "${format}"); """
+ } else {
+ order_qt_sql_tvf """select * from local(
+ "file_path" = "${uri_file}",
+ "backend_id" = "${be_id}",
+ "column_separator"="|",
+ "format" = "${format}") order by k1 limit 10; """
+
+ sql """
+ insert into ${table_name} select * from local(
+ "file_path" = "${uri_file}",
+ "backend_id" = "${be_id}",
+ "column_separator"="|",
+ "format" = "${format}"); """
+ }
+ // where to filter different format data
+ qt_select_doris """ select * from ${table_name} where k1 IS NOT NULL order by k1 limit 10; """
+ }
+ def load_from_s3 = {table_name, uri_file, format ->
+ if (format == "csv") {
+ order_qt_sql_s3 """select * from s3(
+ "uri" = "${uri_file}",
+ "s3.access_key"= "${ak}",
+ "s3.secret_key" = "${sk}",
+ "format" = "${format}",
+ "column_separator"="|",
+ "read_json_by_line"="true") order by c1 limit 10; """
+
+ sql """
+ insert into ${table_name} select * from s3(
+ "uri" = "${uri_file}",
+ "s3.access_key"= "${ak}",
+ "s3.secret_key" = "${sk}",
+ "format" = "${format}",
+ "column_separator"="|",
+ "read_json_by_line"="true"); """
+ } else {
+ order_qt_sql_s3 """select * from s3(
+ "uri" = "${uri_file}",
+ "s3.access_key"= "${ak}",
+ "s3.secret_key" = "${sk}",
+ "format" = "${format}",
+ "read_json_by_line"="true") order by k1 limit 10; """
+
+ sql """
+ insert into ${table_name} select * from s3(
+ "uri" = "${uri_file}",
+ "s3.access_key"= "${ak}",
+ "s3.secret_key" = "${sk}",
+ "format" = "${format}",
+ "read_json_by_line"="true"); """
+ }
+ // where to filter different format data
+ qt_select_doris """ select * from ${table_name} where k1 IS NOT NULL order by k1 limit 10; """
+ }
+
+ // step1. create table
+ // step2. load from s3
+ // step 2.1 format: parquet|orc|json|csv
+ // step3. select *
+ // step4. select element_at(column in first, -1(last), null, 0)
+ // step5. select * from table where element_at(column) equals expr just ok
+ // step6. select * from table where groupby|orderby column
+ // step7. select * from table where groupby|orderby element_at(column)
+
+ def format_order = [
+ "parquet", "orc",
+// "json",
+ "csv"]
+ // create tables
+ for (int i = 0; i < table_names.size(); ++i) {
+ sql """ DROP TABLE IF EXISTS ${table_names[i]} """
+ String result = create_table_with_nested_type(1, [i], table_names[i])
+ sql result
+ }
+
+ //========================= ARRAY =========================
+ // insert into doris table
+ ArrayList<String> array_files = [
+ "${dataFilePath}/one_level_array.parquet",
+ "${dataFilePath}/one_level_array.orc",
+// "${dataFilePath}/one_level_array.json",
+ "${dataFilePath}/one_level_array.csv"
+ ]
+ int fi = 0
+ for (String f : array_files) {
+ sql "truncate table ${table_names[2]};"
Review Comment:
table_names[0]
##########
be/src/vec/data_types/serde/data_type_struct_serde.cpp:
##########
@@ -94,106 +93,125 @@ Status DataTypeStructSerDe::deserialize_one_cell_from_json(IColumn& column, Slic
}
return Status::OK();
}
-
- ReadBuffer rb(slice.data, slice.size);
- ++rb.position();
+ // remove '{' '}'
+ slice.remove_prefix(1);
+ slice.remove_suffix(1);
+ slice.trim_prefix();
bool is_explicit_names = false;
- std::vector<std::string> field_names;
- std::vector<ReadBuffer> field_rbs;
- std::vector<size_t> field_pos;
-
- while (!rb.eof()) {
- StringRef slot(rb.position(), rb.count());
- bool has_quota = false;
- bool is_name = false;
- if (!next_slot_from_string(rb, slot, is_name, has_quota)) {
- return Status::InvalidArgument("Cannot read struct field from text '{}'",
- slot.to_string());
- }
- if (is_name) {
- std::string name = slot.to_string();
- if (!next_slot_from_string(rb, slot, is_name, has_quota)) {
- return Status::InvalidArgument("Cannot read struct field from text '{}'",
- slot.to_string());
- }
- ReadBuffer field_rb(const_cast<char*>(slot.data), slot.size);
- field_names.push_back(name);
- field_rbs.push_back(field_rb);
-
- if (!is_explicit_names) {
- is_explicit_names = true;
+ int nested_level = 0;
+ bool has_quote = false;
+ int start_pos = 0;
+ size_t slice_size = slice.size;
+ bool key_added = false;
+ int idx = 0;
+ char quote_char = 0;
+
+ auto elem_size = elemSerDeSPtrs.size();
+ int field_pos = 0;
+
+ for (; idx < slice_size; ++idx) {
+ char c = slice[idx];
+ if (c == '"' || c == '\'') {
+ if (!has_quote) {
+ quote_char = c;
+ has_quote = !has_quote;
+ } else if (has_quote && quote_char == c) {
+ quote_char = 0;
+ has_quote = !has_quote;
}
- } else {
- ReadBuffer field_rb(const_cast<char*>(slot.data), slot.size);
- field_rbs.push_back(field_rb);
- }
- }
-
- // TODO: should we support insert default field value when actual field number is less than
- // schema field number?
- if (field_rbs.size() != elemSerDeSPtrs.size()) {
- std::string cmp_str = field_rbs.size() > elemSerDeSPtrs.size() ? "more" : "less";
- return Status::InvalidArgument(
- "Actual struct field number {} is {} than schema field number {}.",
- field_rbs.size(), cmp_str, elemSerDeSPtrs.size());
- }
-
- if (is_explicit_names) {
- if (field_names.size() != field_rbs.size()) {
- return Status::InvalidArgument(
- "Struct field name number {} is not equal to field number {}.",
- field_names.size(), field_rbs.size());
- }
- std::unordered_set<std::string> name_set;
- for (size_t i = 0; i < field_names.size(); i++) {
- // check duplicate fields
- auto ret = name_set.insert(field_names[i]);
- if (!ret.second) {
- return Status::InvalidArgument("Struct field name {} is duplicate with others.",
- field_names[i]);
+ } else if (c == '\\' && idx + 1 < slice_size) { //escaped
+ ++idx;
+ } else if (!has_quote && (c == '[' || c == '{')) {
+ ++nested_level;
+ } else if (!has_quote && (c == ']' || c == '}')) {
+ --nested_level;
+ } else if (!has_quote && nested_level == 0 && c == options.map_key_delim && !key_added) {
Review Comment:
what's map_key_delim?
##########
regression-test/suites/datatype_p0/nested_types/base_cases/one_level_nestedtypes_with_s3data.groovy:
##########
@@ -0,0 +1,283 @@
+import org.apache.commons.lang3.StringUtils
+
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("one_level_nestedtypes_with_s3data") {
+ sql """set enable_nereids_planner=false"""
+ sql """ set enable_fallback_to_original_planner=true;"""
+ // this test case aim to test one-level nested type with s3 data
+
+
+ String ak = getS3AK()
+ String sk = getS3SK()
+ String s3_endpoint = getS3Endpoint()
+ String bucket = context.config.otherConfigs.get("s3BucketName");
+
+
+ def dataFilePath = "https://"+"${bucket}"+"."+"${s3_endpoint}"+"/regression/datalake"
+// def dataFilePath = "/mnt/disk1/wangqiannan/export/ol"
+ def table_names = ["test_array_one_level", "test_map_one_level", "test_struct_one_level"]
+
+ def colNameArr = ["c_bool", "c_tinyint", "c_smallint", "c_int", "c_bigint", "c_largeint", "c_float",
Review Comment:
colNames
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [FIX](complextype)fix struct nested complex collection type and and regresstest [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26973:
URL: https://github.com/apache/doris/pull/26973#issuecomment-1818411146
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit 4dd83dd39e20ac1527c03a996abe0009b2a8d391, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4927 4681 4666 4666
q2 364 150 159 150
q3 2057 1936 1875 1875
q4 1409 1247 1238 1238
q5 3970 3965 4017 3965
q6 248 133 133 133
q7 1422 891 910 891
q8 2791 2818 2814 2814
q9 9762 9686 9603 9603
q10 3510 3568 3545 3545
q11 380 250 247 247
q12 449 298 296 296
q13 4596 3831 3833 3831
q14 323 285 297 285
q15 593 533 544 533
q16 663 588 585 585
q17 1152 981 976 976
q18 7726 7378 7404 7378
q19 1702 1699 1668 1668
q20 562 307 290 290
q21 4399 3985 4026 3985
q22 471 371 362 362
Total cold run time: 53476 ms
Total hot run time: 49316 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4600 4600 4591 4591
q2 340 229 273 229
q3 4004 3983 4001 3983
q4 2731 2704 2695 2695
q5 9670 9513 9631 9513
q6 253 131 133 131
q7 3040 2501 2489 2489
q8 4465 4456 4482 4456
q9 13221 13085 13232 13085
q10 4138 4220 4186 4186
q11 781 705 683 683
q12 985 830 810 810
q13 4306 3626 3547 3547
q14 415 356 352 352
q15 582 533 547 533
q16 739 661 682 661
q17 3869 3800 3870 3800
q18 9455 8987 9133 8987
q19 1845 1815 1797 1797
q20 2402 2083 2045 2045
q21 8777 8632 8657 8632
q22 926 800 808 800
Total cold run time: 81544 ms
Total hot run time: 78005 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [FIX](complextype)fix struct nested complex collection type and and regresstest [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26973:
URL: https://github.com/apache/doris/pull/26973#issuecomment-1818617433
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [FIX](complextype)fix struct nested complex collection type and and regresstest [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26973:
URL: https://github.com/apache/doris/pull/26973#issuecomment-1813984843
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.19 seconds
stream load tsv: 567 seconds loaded 74807831229 Bytes, about 125 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17100787575 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [FIX](complextype)fix struct nested complex collection type and and regresstest [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26973:
URL: https://github.com/apache/doris/pull/26973#issuecomment-1818365862
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [FIX](complextype)fix struct nested complex collection type and and regresstest [doris]
Posted by "amorynan (via GitHub)" <gi...@apache.org>.
amorynan commented on PR #26973:
URL: https://github.com/apache/doris/pull/26973#issuecomment-1818455803
run pipelinex_p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [FIX](complextype)fix struct nested complex collection type and and regresstest [doris]
Posted by "amorynan (via GitHub)" <gi...@apache.org>.
amorynan commented on PR #26973:
URL: https://github.com/apache/doris/pull/26973#issuecomment-1818179807
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [FIX](complextype)fix struct nested complex collection type and and regresstest [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26973:
URL: https://github.com/apache/doris/pull/26973#issuecomment-1818425515
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.74 seconds
stream load tsv: 569 seconds loaded 74807831229 Bytes, about 125 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17099010030 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [FIX](complextype)fix struct nested complex collection type and and regresstest [doris]
Posted by "amorynan (via GitHub)" <gi...@apache.org>.
amorynan commented on PR #26973:
URL: https://github.com/apache/doris/pull/26973#issuecomment-1815684688
run p1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [FIX](complextype)fix struct nested complex collection type and and regresstest [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26973:
URL: https://github.com/apache/doris/pull/26973#issuecomment-1814465040
TeamCity be ut coverage result:
Function Coverage: 36.75% (8410/22883)
Line Coverage: 29.27% (68398/233665)
Region Coverage: 27.86% (35356/126925)
Branch Coverage: 24.61% (18066/73408)
Coverage Report: http://coverage.selectdb-in.cc/coverage/6acef23b0e81d666c381128fbff51b4d85ebcc62_6acef23b0e81d666c381128fbff51b4d85ebcc62/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org