You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "amorynan (via GitHub)" <gi...@apache.org> on 2023/11/14 10:36:47 UTC

[PR] [FIX](complextype)fix struct nested complex collection type and and regresstest [doris]

amorynan opened a new pull request, #26973:
URL: https://github.com/apache/doris/pull/26973

   ## Proposed changes
   this pr do 
   1. fix array/map/struct write next_rowid for num_rows when we read from segmentfile
   2.improve struct can nested complex type for load action
   3.add one/two-level regress test for complex types
   
   Issue Number: close #xxx
   
   <!--Describe your changes.-->
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [FIX](complextype)fix struct nested complex collection type and and regresstest [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26973:
URL: https://github.com/apache/doris/pull/26973#issuecomment-1818183580

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [FIX](complextype)fix struct nested complex collection type and and regresstest [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26973:
URL: https://github.com/apache/doris/pull/26973#issuecomment-1818236409

   TeamCity be ut coverage result:
    Function Coverage: 36.51% (8450/23143) 
    Line Coverage: 28.87% (68687/237935)
    Region Coverage: 27.83% (35521/127623)
    Branch Coverage: 24.58% (18111/73670)
    Coverage Report: http://coverage.selectdb-in.cc/coverage/16a3ed8660cb5b8fcd42b337d273da7f554d721c_16a3ed8660cb5b8fcd42b337d273da7f554d721c/report/index.html


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [FIX](complextype)fix struct nested complex collection type and and regresstest [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26973:
URL: https://github.com/apache/doris/pull/26973#issuecomment-1818396075

   TeamCity be ut coverage result:
    Function Coverage: 36.53% (8450/23134) 
    Line Coverage: 28.88% (68692/237888)
    Region Coverage: 27.84% (35524/127607)
    Branch Coverage: 24.59% (18113/73662)
    Coverage Report: http://coverage.selectdb-in.cc/coverage/4dd83dd39e20ac1527c03a996abe0009b2a8d391_4dd83dd39e20ac1527c03a996abe0009b2a8d391/report/index.html


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [FIX](complextype)fix struct nested complex collection type and and regresstest [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26973:
URL: https://github.com/apache/doris/pull/26973#issuecomment-1814017176

   
   <details>
   <summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
   
   ```
   Tpch sf100 test result on commit 18caea5bad584f6fff71d1da6ef0e045e54aa311, data reload: false
   
   run tpch-sf100 query with default conf and session variables
   q1	4997	4654	4605	4605
   q2	351	137	137	137
   q3	2034	1858	1839	1839
   q4	1392	1291	1267	1267
   q5	4006	4005	4061	4005
   q6	241	130	135	130
   q7	1406	884	899	884
   q8	2765	2807	2786	2786
   q9	9685	9585	9396	9396
   q10	3498	3583	3576	3576
   q11	380	256	249	249
   q12	446	296	298	296
   q13	4583	3796	3824	3796
   q14	321	290	286	286
   q15	579	526	519	519
   q16	674	585	588	585
   q17	1141	984	982	982
   q18	7816	7471	7362	7362
   q19	1678	1703	1694	1694
   q20	535	328	290	290
   q21	4437	3990	3978	3978
   q22	486	378	381	378
   Total cold run time: 53451 ms
   Total hot run time: 49040 ms
   
   run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
   q1	4615	4580	4588	4580
   q2	341	230	265	230
   q3	4031	4024	4022	4022
   q4	2718	2703	2708	2703
   q5	9709	9725	9678	9678
   q6	241	127	126	126
   q7	3030	2522	2482	2482
   q8	4393	4475	4431	4431
   q9	12933	12907	12849	12849
   q10	4134	4184	4203	4184
   q11	812	673	647	647
   q12	990	816	795	795
   q13	4322	3545	3565	3545
   q14	378	342	346	342
   q15	575	529	517	517
   q16	774	658	691	658
   q17	3815	3885	3901	3885
   q18	9643	9056	8896	8896
   q19	1836	1828	1786	1786
   q20	2422	2042	2034	2034
   q21	8783	8544	8439	8439
   q22	949	851	846	846
   Total cold run time: 81444 ms
   Total hot run time: 77675 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [FIX](complextype)fix struct nested complex collection type and and regresstest [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26973:
URL: https://github.com/apache/doris/pull/26973#issuecomment-1811986876

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [FIX](complextype)fix struct nested complex collection type and and regresstest [doris]

Posted by "amorynan (via GitHub)" <gi...@apache.org>.
amorynan commented on PR #26973:
URL: https://github.com/apache/doris/pull/26973#issuecomment-1818407448

   run pipelinex_p0


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [FIX](complextype)fix struct nested complex collection type and and regresstest [doris]

Posted by "amorynan (via GitHub)" <gi...@apache.org>.
amorynan commented on PR #26973:
URL: https://github.com/apache/doris/pull/26973#issuecomment-1818358547

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [FIX](complextype)fix struct nested complex collection type and and regresstest [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26973:
URL: https://github.com/apache/doris/pull/26973#issuecomment-1813865446

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [FIX](complextype)fix struct nested complex collection type and and regresstest [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26973:
URL: https://github.com/apache/doris/pull/26973#issuecomment-1813936754

   TeamCity be ut coverage result:
    Function Coverage: 36.78% (8410/22868) 
    Line Coverage: 29.29% (68415/233563)
    Region Coverage: 27.86% (35345/126849)
    Branch Coverage: 24.61% (18059/73374)
    Coverage Report: http://coverage.selectdb-in.cc/coverage/18caea5bad584f6fff71d1da6ef0e045e54aa311_18caea5bad584f6fff71d1da6ef0e045e54aa311/report/index.html


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [FIX](complextype)fix struct nested complex collection type and and regresstest [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26973:
URL: https://github.com/apache/doris/pull/26973#issuecomment-1818617550

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [FIX](complextype)fix struct nested complex collection type and and regresstest [doris]

Posted by "amorynan (via GitHub)" <gi...@apache.org>.
amorynan commented on PR #26973:
URL: https://github.com/apache/doris/pull/26973#issuecomment-1815642044

   run pipelinex_p0


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [FIX](complextype)fix struct nested complex collection type and and regresstest [doris]

Posted by "xiaokang (via GitHub)" <gi...@apache.org>.
xiaokang merged PR #26973:
URL: https://github.com/apache/doris/pull/26973


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [FIX](complextype)fix struct nested complex collection type and and regresstest [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26973:
URL: https://github.com/apache/doris/pull/26973#issuecomment-1818203279

   
   <details>
   <summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
   
   ```
   Tpch sf100 test result on commit 16a3ed8660cb5b8fcd42b337d273da7f554d721c, data reload: false
   
   run tpch-sf100 query with default conf and session variables
   q1	4932	4683	4687	4683
   q2	356	161	159	159
   q3	2034	1950	1847	1847
   q4	1379	1289	1223	1223
   q5	3934	3959	4015	3959
   q6	252	126	130	126
   q7	1392	878	894	878
   q8	2780	2771	2760	2760
   q9	9985	9451	9572	9451
   q10	3467	3512	3523	3512
   q11	381	250	249	249
   q12	440	296	293	293
   q13	4564	3819	3808	3808
   q14	316	286	297	286
   q15	584	526	522	522
   q16	674	578	582	578
   q17	1126	975	942	942
   q18	7915	7560	7527	7527
   q19	1660	1670	1657	1657
   q20	571	294	289	289
   q21	4413	3933	4017	3933
   q22	475	367	374	367
   Total cold run time: 53630 ms
   Total hot run time: 49049 ms
   
   run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
   q1	4609	4558	4577	4558
   q2	348	230	263	230
   q3	4010	4000	3999	3999
   q4	2719	2711	2711	2711
   q5	9602	9576	9616	9576
   q6	252	123	124	123
   q7	3021	2464	2463	2463
   q8	4451	4454	4447	4447
   q9	12892	12851	12866	12851
   q10	4054	4161	4158	4158
   q11	795	664	655	655
   q12	976	833	818	818
   q13	4273	3595	3559	3559
   q14	379	345	342	342
   q15	571	517	523	517
   q16	736	692	662	662
   q17	3914	3894	3934	3894
   q18	9623	9217	9097	9097
   q19	1804	1767	1781	1767
   q20	2416	2089	2048	2048
   q21	8839	8845	8724	8724
   q22	878	802	788	788
   Total cold run time: 81162 ms
   Total hot run time: 77987 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [FIX](complextype)fix struct nested complex collection type and and regresstest [doris]

Posted by "amorynan (via GitHub)" <gi...@apache.org>.
amorynan commented on PR #26973:
URL: https://github.com/apache/doris/pull/26973#issuecomment-1813860209

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [FIX](complextype)fix struct nested complex collection type and and regresstest [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26973:
URL: https://github.com/apache/doris/pull/26973#issuecomment-1809960360

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [FIX](complextype)fix struct nested complex collection type and and regresstest [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26973:
URL: https://github.com/apache/doris/pull/26973#issuecomment-1818207196

   (From new machine)TeamCity pipeline, clickbench performance test result:
    the sum of best hot time: 44.67 seconds
    stream load tsv:          566 seconds loaded 74807831229 Bytes, about 126 MB/s
    stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
    stream load orc:          65 seconds loaded 1101869774 Bytes, about 16 MB/s
    stream load parquet:          32 seconds loaded 861443392 Bytes, about 25 MB/s
    insert into select:          28.6 seconds inserted 10000000 Rows, about 349K ops/s
    storage size: 17100218325 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [FIX](complextype)fix struct nested complex collection type and and regresstest [doris]

Posted by "amorynan (via GitHub)" <gi...@apache.org>.
amorynan commented on PR #26973:
URL: https://github.com/apache/doris/pull/26973#issuecomment-1814423237

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [FIX](complextype)fix struct nested complex collection type and and regresstest [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26973:
URL: https://github.com/apache/doris/pull/26973#issuecomment-1814673079

   (From new machine)TeamCity pipeline, clickbench performance test result:
    the sum of best hot time: 45.45 seconds
    stream load tsv:          566 seconds loaded 74807831229 Bytes, about 126 MB/s
    stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
    stream load orc:          65 seconds loaded 1101869774 Bytes, about 16 MB/s
    stream load parquet:          32 seconds loaded 861443392 Bytes, about 25 MB/s
    insert into select:          28.9 seconds inserted 10000000 Rows, about 346K ops/s
    storage size: 17098963746 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [FIX](complextype)fix struct nested complex collection type and and regresstest [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26973:
URL: https://github.com/apache/doris/pull/26973#issuecomment-1813865402

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [FIX](complextype)fix struct nested complex collection type and and regresstest [doris]

Posted by "amorynan (via GitHub)" <gi...@apache.org>.
amorynan commented on code in PR #26973:
URL: https://github.com/apache/doris/pull/26973#discussion_r1398630717


##########
be/src/vec/data_types/serde/data_type_struct_serde.cpp:
##########
@@ -94,106 +93,125 @@ Status DataTypeStructSerDe::deserialize_one_cell_from_json(IColumn& column, Slic
         }
         return Status::OK();
     }
-
-    ReadBuffer rb(slice.data, slice.size);
-    ++rb.position();
+    // remove '{' '}'
+    slice.remove_prefix(1);
+    slice.remove_suffix(1);
+    slice.trim_prefix();
 
     bool is_explicit_names = false;
-    std::vector<std::string> field_names;
-    std::vector<ReadBuffer> field_rbs;
-    std::vector<size_t> field_pos;
-
-    while (!rb.eof()) {
-        StringRef slot(rb.position(), rb.count());
-        bool has_quota = false;
-        bool is_name = false;
-        if (!next_slot_from_string(rb, slot, is_name, has_quota)) {
-            return Status::InvalidArgument("Cannot read struct field from text '{}'",
-                                           slot.to_string());
-        }
-        if (is_name) {
-            std::string name = slot.to_string();
-            if (!next_slot_from_string(rb, slot, is_name, has_quota)) {
-                return Status::InvalidArgument("Cannot read struct field from text '{}'",
-                                               slot.to_string());
-            }
-            ReadBuffer field_rb(const_cast<char*>(slot.data), slot.size);
-            field_names.push_back(name);
-            field_rbs.push_back(field_rb);
-
-            if (!is_explicit_names) {
-                is_explicit_names = true;
+    int nested_level = 0;
+    bool has_quote = false;
+    int start_pos = 0;
+    size_t slice_size = slice.size;
+    bool key_added = false;
+    int idx = 0;
+    char quote_char = 0;
+
+    auto elem_size = elemSerDeSPtrs.size();
+    int field_pos = 0;
+
+    for (; idx < slice_size; ++idx) {
+        char c = slice[idx];
+        if (c == '"' || c == '\'') {
+            if (!has_quote) {
+                quote_char = c;
+                has_quote = !has_quote;
+            } else if (has_quote && quote_char == c) {
+                quote_char = 0;
+                has_quote = !has_quote;
             }
-        } else {
-            ReadBuffer field_rb(const_cast<char*>(slot.data), slot.size);
-            field_rbs.push_back(field_rb);
-        }
-    }
-
-    // TODO: should we support insert default field value when actual field number is less than
-    // schema field number?
-    if (field_rbs.size() != elemSerDeSPtrs.size()) {
-        std::string cmp_str = field_rbs.size() > elemSerDeSPtrs.size() ? "more" : "less";
-        return Status::InvalidArgument(
-                "Actual struct field number {} is {} than schema field number {}.",
-                field_rbs.size(), cmp_str, elemSerDeSPtrs.size());
-    }
-
-    if (is_explicit_names) {
-        if (field_names.size() != field_rbs.size()) {
-            return Status::InvalidArgument(
-                    "Struct field name number {} is not equal to field number {}.",
-                    field_names.size(), field_rbs.size());
-        }
-        std::unordered_set<std::string> name_set;
-        for (size_t i = 0; i < field_names.size(); i++) {
-            // check duplicate fields
-            auto ret = name_set.insert(field_names[i]);
-            if (!ret.second) {
-                return Status::InvalidArgument("Struct field name {} is duplicate with others.",
-                                               field_names[i]);
+        } else if (c == '\\' && idx + 1 < slice_size) { //escaped
+            ++idx;
+        } else if (!has_quote && (c == '[' || c == '{')) {
+            ++nested_level;
+        } else if (!has_quote && (c == ']' || c == '}')) {
+            --nested_level;
+        } else if (!has_quote && nested_level == 0 && c == options.map_key_delim && !key_added) {

Review Comment:
   ","



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [FIX](complextype)fix struct nested complex collection type and and regresstest [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26973:
URL: https://github.com/apache/doris/pull/26973#issuecomment-1814432148

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [FIX](complextype)fix struct nested complex collection type and and regresstest [doris]

Posted by "amorynan (via GitHub)" <gi...@apache.org>.
amorynan commented on PR #26973:
URL: https://github.com/apache/doris/pull/26973#issuecomment-1814115932

   run p0


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [FIX](complextype)fix struct nested complex collection type and and regresstest [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26973:
URL: https://github.com/apache/doris/pull/26973#issuecomment-1814665906

   
   <details>
   <summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
   
   ```
   Tpch sf100 test result on commit 6acef23b0e81d666c381128fbff51b4d85ebcc62, data reload: false
   
   run tpch-sf100 query with default conf and session variables
   q1	4940	4719	4675	4675
   q2	354	156	157	156
   q3	2044	1937	1914	1914
   q4	1385	1279	1244	1244
   q5	3963	3972	4082	3972
   q6	251	132	133	132
   q7	1417	866	894	866
   q8	2755	2805	2801	2801
   q9	9860	9788	9577	9577
   q10	3465	3533	3530	3530
   q11	380	249	239	239
   q12	444	307	297	297
   q13	4555	3846	3796	3796
   q14	334	281	296	281
   q15	587	525	526	525
   q16	668	589	590	589
   q17	1127	945	923	923
   q18	7818	7335	7429	7335
   q19	1695	1690	1658	1658
   q20	579	325	295	295
   q21	4443	3951	3980	3951
   q22	487	383	381	381
   Total cold run time: 53551 ms
   Total hot run time: 49137 ms
   
   run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
   q1	4615	4547	4667	4547
   q2	345	228	262	228
   q3	4016	4024	3997	3997
   q4	2718	2697	2702	2697
   q5	9573	9590	9518	9518
   q6	253	125	125	125
   q7	2593	2292	2296	2292
   q8	4438	4397	4496	4397
   q9	13208	13173	13154	13154
   q10	4090	4190	4174	4174
   q11	800	682	704	682
   q12	974	819	810	810
   q13	4280	3578	3549	3549
   q14	384	343	356	343
   q15	581	518	537	518
   q16	738	688	666	666
   q17	3864	3917	3873	3873
   q18	9533	9124	9080	9080
   q19	1798	1804	1785	1785
   q20	2405	2075	2075	2075
   q21	8917	8456	8687	8456
   q22	910	813	813	813
   Total cold run time: 81033 ms
   Total hot run time: 77779 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [FIX](complextype)fix struct nested complex collection type and and regresstest [doris]

Posted by "xiaokang (via GitHub)" <gi...@apache.org>.
xiaokang commented on code in PR #26973:
URL: https://github.com/apache/doris/pull/26973#discussion_r1396690419


##########
regression-test/suites/datatype_p0/nested_types/base_cases/one_level_nestedtypes_with_s3data.groovy:
##########
@@ -0,0 +1,283 @@
+import org.apache.commons.lang3.StringUtils
+
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("one_level_nestedtypes_with_s3data") {
+    sql """set enable_nereids_planner=false"""
+    sql """ set enable_fallback_to_original_planner=true;"""
+    // this test case aim to test one-level nested type with s3 data
+
+
+    String ak = getS3AK()
+    String sk = getS3SK()
+    String s3_endpoint = getS3Endpoint()
+    String bucket = context.config.otherConfigs.get("s3BucketName");
+
+
+    def dataFilePath = "https://"+"${bucket}"+"."+"${s3_endpoint}"+"/regression/datalake"
+//    def dataFilePath = "/mnt/disk1/wangqiannan/export/ol"
+    def table_names = ["test_array_one_level", "test_map_one_level", "test_struct_one_level"]
+
+    def colNameArr = ["c_bool", "c_tinyint", "c_smallint", "c_int", "c_bigint", "c_largeint", "c_float",
+                      "c_double", "c_decimal", "c_decimalv3", "c_date", "c_datetime", "c_datev2", "c_datetimev2",
+                      "c_char", "c_varchar", "c_string"]
+
+    def groupby_or_orderby_exception = {is_groupby, table_name, col_name ->
+        test {
+            if (is_groupby) {
+                sql "select ${col_name} from ${table_name} group by ${col_name};"
+            } else {
+                sql "select ${col_name} from ${table_name} order by ${col_name};"
+            }
+            exception("errCode = 2, detailMessage = Doris hll, bitmap, array, map, struct, jsonb, variant column must use with specific function, and don't support filter, group by or order by")
+        }
+    }
+
+    def groupby_or_orderby_element_at = {is_groupby, table_name, agg_expr ->
+        if (is_groupby) {
+            order_qt_sql "select ${agg_expr} from ${table_name} where k1 IS NOT NULL group by ${agg_expr};"
+        } else {
+            order_qt_sql "select ${agg_expr} from ${table_name} where k1 IS NOT NULL order by ${agg_expr} limit 10;"
+        }
+    }
+
+    def be_id = 10139
+    def load_from_tvf = {table_name, uri_file, format ->
+        if (format == "csv") {
+            order_qt_sql_tvf """select * from local(
+                "file_path" = "${uri_file}",
+                "backend_id" = "${be_id}",
+                "column_separator"="|",
+                "format" = "${format}") order by c1 limit 10; """
+
+            sql """
+            insert into ${table_name} select * from local(
+            "file_path" = "${uri_file}",
+            "backend_id" = "${be_id}",
+            "column_separator"="|",
+            "format" = "${format}"); """
+        } else {
+            order_qt_sql_tvf """select * from local(
+                "file_path" = "${uri_file}",
+                "backend_id" = "${be_id}",
+                "column_separator"="|",
+                "format" = "${format}") order by k1 limit 10; """
+
+            sql """
+            insert into ${table_name} select * from local(
+            "file_path" = "${uri_file}",
+            "backend_id" = "${be_id}",
+            "column_separator"="|",
+            "format" = "${format}"); """
+        }
+        // where to filter different format data
+        qt_select_doris """ select * from ${table_name} where k1 IS NOT NULL order by k1 limit 10; """
+    }
+    def load_from_s3 = {table_name, uri_file, format ->
+        if (format == "csv") {
+            order_qt_sql_s3 """select * from s3(
+                "uri" = "${uri_file}",
+                    "s3.access_key"= "${ak}",
+                    "s3.secret_key" = "${sk}",
+                    "format" = "${format}",
+                    "column_separator"="|",
+                    "read_json_by_line"="true") order by c1 limit 10; """
+
+            sql """
+            insert into ${table_name} select * from s3(
+            "uri" = "${uri_file}",
+                    "s3.access_key"= "${ak}",
+                    "s3.secret_key" = "${sk}",
+                    "format" = "${format}",
+                    "column_separator"="|",
+                    "read_json_by_line"="true"); """
+        } else {
+            order_qt_sql_s3 """select * from s3(
+                "uri" = "${uri_file}",
+                    "s3.access_key"= "${ak}",
+                    "s3.secret_key" = "${sk}",
+                    "format" = "${format}",
+                    "read_json_by_line"="true") order by k1 limit 10; """
+
+            sql """
+            insert into ${table_name} select * from s3(
+            "uri" = "${uri_file}",
+                    "s3.access_key"= "${ak}",
+                    "s3.secret_key" = "${sk}",
+                    "format" = "${format}",
+                    "read_json_by_line"="true"); """
+        }
+        // where to filter different format data
+        qt_select_doris """ select * from ${table_name} where k1 IS NOT NULL order by k1 limit 10; """
+    }
+
+    // step1. create table
+    // step2. load from s3
+    //      step 2.1 format: parquet|orc|json|csv
+    // step3. select *
+    // step4. select element_at(column in first, -1(last), null, 0)
+    // step5. select * from table where element_at(column) equals expr just ok
+    // step6. select * from table where groupby|orderby column
+    // step7. select * from table where groupby|orderby element_at(column)
+
+    def format_order = [
+            "parquet", "orc",
+//            "json",
+            "csv"]
+    // create tables
+    for (int i = 0; i < table_names.size(); ++i) {
+        sql """ DROP TABLE IF EXISTS ${table_names[i]} """
+        String result = create_table_with_nested_type(1, [i], table_names[i])
+        sql result
+    }
+
+    //========================= ARRAY =========================
+    // insert into doris table
+    ArrayList<String> array_files = [
+                                    "${dataFilePath}/one_level_array.parquet",
+                                    "${dataFilePath}/one_level_array.orc",
+//                                    "${dataFilePath}/one_level_array.json",
+                                    "${dataFilePath}/one_level_array.csv"
+    ]
+    int fi = 0
+    for (String f : array_files) {
+        sql "truncate table ${table_names[2]};"
+//        load_from_tvf(table_names[0], f, format_order[fi])
+        load_from_s3(table_names[0], f, format_order[fi])
+        ++ fi
+    }
+    // select element_at(column)
+    for (String col : colNameArr) {
+        // first
+        order_qt_select_arr "select ${col}[1] from ${table_names[0]} where k1 IS NOT NULL order by k1 limit 10;"
+        // last
+        order_qt_select_arr "select ${col}[-1] from ${table_names[0]} where k1 IS NOT NULL order by k1 limit 10;"
+        // null
+        order_qt_select_arr_null "select ${col}[0] from ${table_names[0]} where k1 IS NOT NULL order by k1 limit 10;"
+        // null
+        order_qt_select_arr_null "select ${col}[1000] from ${table_names[0]} where k1 IS NOT NULL order by k1 limit 10;"
+    }
+    // select * from table where element_at(column) with equal expr
+    for (String col : colNameArr) {
+        order_qt_select_arr "select ${col}[1], ${col}[-1] from ${table_names[0]} where k1 IS NOT NULL AND ${col}[1]<${col}[-1] order by k1 limit 10;"
+    }
+    // select * from table where groupby|orderby column will meet exception
+    for (String col : colNameArr) {
+        groupby_or_orderby_exception(true, table_names[0], col)
+        groupby_or_orderby_exception(false, table_names[0], col)
+    }
+    // select * from table where groupby|orderby element_at(column)
+    for (String col : colNameArr) {
+        String agg_expr = "${col}[1]"
+        groupby_or_orderby_element_at(true, table_names[0], agg_expr)
+        groupby_or_orderby_element_at(false, table_names[0], agg_expr)
+    }
+
+    //========================== MAP ==========================
+    // insert into doris table
+    ArrayList<String> map_files = [
+                                    "${dataFilePath}/one_level_map.parquet",
+                                    "${dataFilePath}/one_level_map.orc",
+//                                    "${dataFilePath}/one_level_map.json",
+                                    "${dataFilePath}/one_level_map.csv"
+    ]
+    fi = 0
+    for (String f : map_files) {
+        sql "truncate table ${table_names[2]};"

Review Comment:
   table_names[1]



##########
regression-test/suites/datatype_p0/nested_types/base_cases/one_level_nestedtypes_with_s3data.groovy:
##########
@@ -0,0 +1,283 @@
+import org.apache.commons.lang3.StringUtils
+
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("one_level_nestedtypes_with_s3data") {
+    sql """set enable_nereids_planner=false"""
+    sql """ set enable_fallback_to_original_planner=true;"""
+    // this test case aim to test one-level nested type with s3 data
+
+
+    String ak = getS3AK()
+    String sk = getS3SK()
+    String s3_endpoint = getS3Endpoint()
+    String bucket = context.config.otherConfigs.get("s3BucketName");
+
+
+    def dataFilePath = "https://"+"${bucket}"+"."+"${s3_endpoint}"+"/regression/datalake"
+//    def dataFilePath = "/mnt/disk1/wangqiannan/export/ol"
+    def table_names = ["test_array_one_level", "test_map_one_level", "test_struct_one_level"]
+
+    def colNameArr = ["c_bool", "c_tinyint", "c_smallint", "c_int", "c_bigint", "c_largeint", "c_float",
+                      "c_double", "c_decimal", "c_decimalv3", "c_date", "c_datetime", "c_datev2", "c_datetimev2",
+                      "c_char", "c_varchar", "c_string"]
+
+    def groupby_or_orderby_exception = {is_groupby, table_name, col_name ->
+        test {
+            if (is_groupby) {
+                sql "select ${col_name} from ${table_name} group by ${col_name};"
+            } else {
+                sql "select ${col_name} from ${table_name} order by ${col_name};"
+            }
+            exception("errCode = 2, detailMessage = Doris hll, bitmap, array, map, struct, jsonb, variant column must use with specific function, and don't support filter, group by or order by")
+        }
+    }
+
+    def groupby_or_orderby_element_at = {is_groupby, table_name, agg_expr ->
+        if (is_groupby) {
+            order_qt_sql "select ${agg_expr} from ${table_name} where k1 IS NOT NULL group by ${agg_expr};"
+        } else {
+            order_qt_sql "select ${agg_expr} from ${table_name} where k1 IS NOT NULL order by ${agg_expr} limit 10;"
+        }
+    }
+
+    def be_id = 10139
+    def load_from_tvf = {table_name, uri_file, format ->
+        if (format == "csv") {
+            order_qt_sql_tvf """select * from local(
+                "file_path" = "${uri_file}",
+                "backend_id" = "${be_id}",
+                "column_separator"="|",
+                "format" = "${format}") order by c1 limit 10; """
+
+            sql """
+            insert into ${table_name} select * from local(
+            "file_path" = "${uri_file}",
+            "backend_id" = "${be_id}",
+            "column_separator"="|",
+            "format" = "${format}"); """
+        } else {
+            order_qt_sql_tvf """select * from local(
+                "file_path" = "${uri_file}",
+                "backend_id" = "${be_id}",
+                "column_separator"="|",
+                "format" = "${format}") order by k1 limit 10; """
+
+            sql """
+            insert into ${table_name} select * from local(
+            "file_path" = "${uri_file}",
+            "backend_id" = "${be_id}",
+            "column_separator"="|",
+            "format" = "${format}"); """
+        }
+        // where to filter different format data
+        qt_select_doris """ select * from ${table_name} where k1 IS NOT NULL order by k1 limit 10; """
+    }
+    def load_from_s3 = {table_name, uri_file, format ->
+        if (format == "csv") {
+            order_qt_sql_s3 """select * from s3(
+                "uri" = "${uri_file}",
+                    "s3.access_key"= "${ak}",
+                    "s3.secret_key" = "${sk}",
+                    "format" = "${format}",
+                    "column_separator"="|",
+                    "read_json_by_line"="true") order by c1 limit 10; """
+
+            sql """
+            insert into ${table_name} select * from s3(
+            "uri" = "${uri_file}",
+                    "s3.access_key"= "${ak}",
+                    "s3.secret_key" = "${sk}",
+                    "format" = "${format}",
+                    "column_separator"="|",
+                    "read_json_by_line"="true"); """
+        } else {
+            order_qt_sql_s3 """select * from s3(
+                "uri" = "${uri_file}",
+                    "s3.access_key"= "${ak}",
+                    "s3.secret_key" = "${sk}",
+                    "format" = "${format}",
+                    "read_json_by_line"="true") order by k1 limit 10; """
+
+            sql """
+            insert into ${table_name} select * from s3(
+            "uri" = "${uri_file}",
+                    "s3.access_key"= "${ak}",
+                    "s3.secret_key" = "${sk}",
+                    "format" = "${format}",
+                    "read_json_by_line"="true"); """
+        }
+        // where to filter different format data
+        qt_select_doris """ select * from ${table_name} where k1 IS NOT NULL order by k1 limit 10; """
+    }
+
+    // step1. create table
+    // step2. load from s3
+    //      step 2.1 format: parquet|orc|json|csv
+    // step3. select *
+    // step4. select element_at(column in first, -1(last), null, 0)
+    // step5. select * from table where element_at(column) equals expr just ok
+    // step6. select * from table where groupby|orderby column
+    // step7. select * from table where groupby|orderby element_at(column)
+
+    def format_order = [
+            "parquet", "orc",
+//            "json",
+            "csv"]
+    // create tables
+    for (int i = 0; i < table_names.size(); ++i) {
+        sql """ DROP TABLE IF EXISTS ${table_names[i]} """
+        String result = create_table_with_nested_type(1, [i], table_names[i])
+        sql result
+    }
+
+    //========================= ARRAY =========================
+    // insert into doris table
+    ArrayList<String> array_files = [
+                                    "${dataFilePath}/one_level_array.parquet",
+                                    "${dataFilePath}/one_level_array.orc",
+//                                    "${dataFilePath}/one_level_array.json",
+                                    "${dataFilePath}/one_level_array.csv"
+    ]
+    int fi = 0
+    for (String f : array_files) {
+        sql "truncate table ${table_names[2]};"

Review Comment:
   table_names[0]



##########
be/src/vec/data_types/serde/data_type_struct_serde.cpp:
##########
@@ -94,106 +93,125 @@ Status DataTypeStructSerDe::deserialize_one_cell_from_json(IColumn& column, Slic
         }
         return Status::OK();
     }
-
-    ReadBuffer rb(slice.data, slice.size);
-    ++rb.position();
+    // remove '{' '}'
+    slice.remove_prefix(1);
+    slice.remove_suffix(1);
+    slice.trim_prefix();
 
     bool is_explicit_names = false;
-    std::vector<std::string> field_names;
-    std::vector<ReadBuffer> field_rbs;
-    std::vector<size_t> field_pos;
-
-    while (!rb.eof()) {
-        StringRef slot(rb.position(), rb.count());
-        bool has_quota = false;
-        bool is_name = false;
-        if (!next_slot_from_string(rb, slot, is_name, has_quota)) {
-            return Status::InvalidArgument("Cannot read struct field from text '{}'",
-                                           slot.to_string());
-        }
-        if (is_name) {
-            std::string name = slot.to_string();
-            if (!next_slot_from_string(rb, slot, is_name, has_quota)) {
-                return Status::InvalidArgument("Cannot read struct field from text '{}'",
-                                               slot.to_string());
-            }
-            ReadBuffer field_rb(const_cast<char*>(slot.data), slot.size);
-            field_names.push_back(name);
-            field_rbs.push_back(field_rb);
-
-            if (!is_explicit_names) {
-                is_explicit_names = true;
+    int nested_level = 0;
+    bool has_quote = false;
+    int start_pos = 0;
+    size_t slice_size = slice.size;
+    bool key_added = false;
+    int idx = 0;
+    char quote_char = 0;
+
+    auto elem_size = elemSerDeSPtrs.size();
+    int field_pos = 0;
+
+    for (; idx < slice_size; ++idx) {
+        char c = slice[idx];
+        if (c == '"' || c == '\'') {
+            if (!has_quote) {
+                quote_char = c;
+                has_quote = !has_quote;
+            } else if (has_quote && quote_char == c) {
+                quote_char = 0;
+                has_quote = !has_quote;
             }
-        } else {
-            ReadBuffer field_rb(const_cast<char*>(slot.data), slot.size);
-            field_rbs.push_back(field_rb);
-        }
-    }
-
-    // TODO: should we support insert default field value when actual field number is less than
-    // schema field number?
-    if (field_rbs.size() != elemSerDeSPtrs.size()) {
-        std::string cmp_str = field_rbs.size() > elemSerDeSPtrs.size() ? "more" : "less";
-        return Status::InvalidArgument(
-                "Actual struct field number {} is {} than schema field number {}.",
-                field_rbs.size(), cmp_str, elemSerDeSPtrs.size());
-    }
-
-    if (is_explicit_names) {
-        if (field_names.size() != field_rbs.size()) {
-            return Status::InvalidArgument(
-                    "Struct field name number {} is not equal to field number {}.",
-                    field_names.size(), field_rbs.size());
-        }
-        std::unordered_set<std::string> name_set;
-        for (size_t i = 0; i < field_names.size(); i++) {
-            // check duplicate fields
-            auto ret = name_set.insert(field_names[i]);
-            if (!ret.second) {
-                return Status::InvalidArgument("Struct field name {} is duplicate with others.",
-                                               field_names[i]);
+        } else if (c == '\\' && idx + 1 < slice_size) { //escaped
+            ++idx;
+        } else if (!has_quote && (c == '[' || c == '{')) {
+            ++nested_level;
+        } else if (!has_quote && (c == ']' || c == '}')) {
+            --nested_level;
+        } else if (!has_quote && nested_level == 0 && c == options.map_key_delim && !key_added) {

Review Comment:
   what's map_key_delim?



##########
regression-test/suites/datatype_p0/nested_types/base_cases/one_level_nestedtypes_with_s3data.groovy:
##########
@@ -0,0 +1,283 @@
+import org.apache.commons.lang3.StringUtils
+
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("one_level_nestedtypes_with_s3data") {
+    sql """set enable_nereids_planner=false"""
+    sql """ set enable_fallback_to_original_planner=true;"""
+    // this test case aim to test one-level nested type with s3 data
+
+
+    String ak = getS3AK()
+    String sk = getS3SK()
+    String s3_endpoint = getS3Endpoint()
+    String bucket = context.config.otherConfigs.get("s3BucketName");
+
+
+    def dataFilePath = "https://"+"${bucket}"+"."+"${s3_endpoint}"+"/regression/datalake"
+//    def dataFilePath = "/mnt/disk1/wangqiannan/export/ol"
+    def table_names = ["test_array_one_level", "test_map_one_level", "test_struct_one_level"]
+
+    def colNameArr = ["c_bool", "c_tinyint", "c_smallint", "c_int", "c_bigint", "c_largeint", "c_float",

Review Comment:
   colNames



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [FIX](complextype)fix struct nested complex collection type and and regresstest [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26973:
URL: https://github.com/apache/doris/pull/26973#issuecomment-1818411146

   
   <details>
   <summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
   
   ```
   Tpch sf100 test result on commit 4dd83dd39e20ac1527c03a996abe0009b2a8d391, data reload: false
   
   run tpch-sf100 query with default conf and session variables
   q1	4927	4681	4666	4666
   q2	364	150	159	150
   q3	2057	1936	1875	1875
   q4	1409	1247	1238	1238
   q5	3970	3965	4017	3965
   q6	248	133	133	133
   q7	1422	891	910	891
   q8	2791	2818	2814	2814
   q9	9762	9686	9603	9603
   q10	3510	3568	3545	3545
   q11	380	250	247	247
   q12	449	298	296	296
   q13	4596	3831	3833	3831
   q14	323	285	297	285
   q15	593	533	544	533
   q16	663	588	585	585
   q17	1152	981	976	976
   q18	7726	7378	7404	7378
   q19	1702	1699	1668	1668
   q20	562	307	290	290
   q21	4399	3985	4026	3985
   q22	471	371	362	362
   Total cold run time: 53476 ms
   Total hot run time: 49316 ms
   
   run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
   q1	4600	4600	4591	4591
   q2	340	229	273	229
   q3	4004	3983	4001	3983
   q4	2731	2704	2695	2695
   q5	9670	9513	9631	9513
   q6	253	131	133	131
   q7	3040	2501	2489	2489
   q8	4465	4456	4482	4456
   q9	13221	13085	13232	13085
   q10	4138	4220	4186	4186
   q11	781	705	683	683
   q12	985	830	810	810
   q13	4306	3626	3547	3547
   q14	415	356	352	352
   q15	582	533	547	533
   q16	739	661	682	661
   q17	3869	3800	3870	3800
   q18	9455	8987	9133	8987
   q19	1845	1815	1797	1797
   q20	2402	2083	2045	2045
   q21	8777	8632	8657	8632
   q22	926	800	808	800
   Total cold run time: 81544 ms
   Total hot run time: 78005 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [FIX](complextype)fix struct nested complex collection type and and regresstest [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26973:
URL: https://github.com/apache/doris/pull/26973#issuecomment-1818617433

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [FIX](complextype)fix struct nested complex collection type and and regresstest [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26973:
URL: https://github.com/apache/doris/pull/26973#issuecomment-1813984843

   (From new machine)TeamCity pipeline, clickbench performance test result:
    the sum of best hot time: 45.19 seconds
    stream load tsv:          567 seconds loaded 74807831229 Bytes, about 125 MB/s
    stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
    stream load orc:          65 seconds loaded 1101869774 Bytes, about 16 MB/s
    stream load parquet:          32 seconds loaded 861443392 Bytes, about 25 MB/s
    insert into select:          28.7 seconds inserted 10000000 Rows, about 348K ops/s
    storage size: 17100787575 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [FIX](complextype)fix struct nested complex collection type and and regresstest [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26973:
URL: https://github.com/apache/doris/pull/26973#issuecomment-1818365862

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [FIX](complextype)fix struct nested complex collection type and and regresstest [doris]

Posted by "amorynan (via GitHub)" <gi...@apache.org>.
amorynan commented on PR #26973:
URL: https://github.com/apache/doris/pull/26973#issuecomment-1818455803

   run pipelinex_p0


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [FIX](complextype)fix struct nested complex collection type and and regresstest [doris]

Posted by "amorynan (via GitHub)" <gi...@apache.org>.
amorynan commented on PR #26973:
URL: https://github.com/apache/doris/pull/26973#issuecomment-1818179807

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [FIX](complextype)fix struct nested complex collection type and and regresstest [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26973:
URL: https://github.com/apache/doris/pull/26973#issuecomment-1818425515

   (From new machine)TeamCity pipeline, clickbench performance test result:
    the sum of best hot time: 44.74 seconds
    stream load tsv:          569 seconds loaded 74807831229 Bytes, about 125 MB/s
    stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
    stream load orc:          65 seconds loaded 1101869774 Bytes, about 16 MB/s
    stream load parquet:          32 seconds loaded 861443392 Bytes, about 25 MB/s
    insert into select:          28.8 seconds inserted 10000000 Rows, about 347K ops/s
    storage size: 17099010030 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [FIX](complextype)fix struct nested complex collection type and and regresstest [doris]

Posted by "amorynan (via GitHub)" <gi...@apache.org>.
amorynan commented on PR #26973:
URL: https://github.com/apache/doris/pull/26973#issuecomment-1815684688

   run p1


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [FIX](complextype)fix struct nested complex collection type and and regresstest [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26973:
URL: https://github.com/apache/doris/pull/26973#issuecomment-1814465040

   TeamCity be ut coverage result:
    Function Coverage: 36.75% (8410/22883) 
    Line Coverage: 29.27% (68398/233665)
    Region Coverage: 27.86% (35356/126925)
    Branch Coverage: 24.61% (18066/73408)
    Coverage Report: http://coverage.selectdb-in.cc/coverage/6acef23b0e81d666c381128fbff51b4d85ebcc62_6acef23b0e81d666c381128fbff51b4d85ebcc62/report/index.html


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org