You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "mrhhsg (via GitHub)" <gi...@apache.org> on 2023/11/01 02:38:22 UTC

[PR] [refactor](storage) Expressing the types of computation layer and storage layer in PrimitiveTypeTraits [doris]

mrhhsg opened a new pull request, #26191:
URL: https://github.com/apache/doris/pull/26191

   ## Proposed changes
   
   Issue Number: close #xxx
   
   <!--Describe your changes.-->
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](storage) Expressing the types of computation layer and storage layer in PrimitiveTypeTraits [doris]

Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #26191:
URL: https://github.com/apache/doris/pull/26191#issuecomment-1791676407

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](storage) Expressing the types of computation layer and storage layer in PrimitiveTypeTraits [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26191:
URL: https://github.com/apache/doris/pull/26191#issuecomment-1811912204

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](storage) Expressing the types of computation layer and storage layer in PrimitiveTypeTraits [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26191:
URL: https://github.com/apache/doris/pull/26191#issuecomment-1812097035

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](storage) Expressing the types of computation layer and storage layer in PrimitiveTypeTraits [doris]

Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #26191:
URL: https://github.com/apache/doris/pull/26191#discussion_r1378865207


##########
be/src/olap/in_list_predicate.h:
##########
@@ -395,6 +371,16 @@ class InListPredicateBase : public ColumnPredicate {
                     if (bf->test_bytes(reinterpret_cast<const char*>(value), sizeof(uint24_t))) {
                         return true;
                     }
+                } else if constexpr (Type == PrimitiveType::TYPE_DECIMALV2) {
+                    // DecimalV2 using decimal12_t in bloom filter in storage layer,
+                    // should convert value to decimal12_t
+                    const T* value = (const T*)(iter->get_value());
+                    decimal12_t decimal12_t_val(value->int_value(), value->frac_value());
+                    if (bf->test_bytes(
+                                const_cast<char*>(reinterpret_cast<const char*>(&decimal12_t_val)),
+                                sizeof(decimal12_t))) {
+                        return true;

Review Comment:
   what about datev1 and datetimev1



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](storage) Expressing the types of computation layer and storage layer in PrimitiveTypeTraits [doris]

Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #26191:
URL: https://github.com/apache/doris/pull/26191#issuecomment-1809340662

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](storage) Expressing the types of computation layer and storage layer in PrimitiveTypeTraits [doris]

Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #26191:
URL: https://github.com/apache/doris/pull/26191#issuecomment-1811910306

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](storage) Expressing the types of computation layer and storage layer in PrimitiveTypeTraits [doris]

Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #26191:
URL: https://github.com/apache/doris/pull/26191#discussion_r1378874597


##########
regression-test/suites/datatype_p0/scalar_types/load.groovy:
##########
@@ -90,8 +90,8 @@ suite("test_scalar_types_load", "p0") {
             `c_float` float NULL,
             `c_double` double NULL,
             `c_decimal` decimal(20, 3) NULL,
-            `c_date` date NULL,
-            `c_datetime` datetime NULL,
+            `c_date` datev1 NULL,
+            `c_datetime` datetimev1 NULL,
             `c_datev2` datev2 NULL,

Review Comment:
   we should add more test cases including:
   1. in list predicate
   2. delete predicate
   3. comparison predicate
   4. is null predicate
   
   test index:
   bloom filter index
   min max index
   bitmap index
   
   test key range
   
   test all data types
   
   test runtime filter // maybe very difficult



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](storage) Expressing the types of computation layer and storage layer in PrimitiveTypeTraits [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26191:
URL: https://github.com/apache/doris/pull/26191#issuecomment-1788529915

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](storage) Expressing the types of computation layer and storage layer in PrimitiveTypeTraits [doris]

Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #26191:
URL: https://github.com/apache/doris/pull/26191#issuecomment-1788522388

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](storage) Expressing the types of computation layer and storage layer in PrimitiveTypeTraits [doris]

Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #26191:
URL: https://github.com/apache/doris/pull/26191#issuecomment-1809312759

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](storage) Expressing the types of computation layer and storage layer in PrimitiveTypeTraits [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26191:
URL: https://github.com/apache/doris/pull/26191#issuecomment-1788318333

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](storage) Expressing the types of computation layer and storage layer in PrimitiveTypeTraits [doris]

Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #26191:
URL: https://github.com/apache/doris/pull/26191#discussion_r1378861254


##########
be/src/olap/comparison_predicate.h:
##########
@@ -232,8 +251,16 @@ class ComparisonPredicateBase : public ColumnPredicate {
             if constexpr (std::is_same_v<T, StringRef>) {
                 return bf->test_bytes(_value.data, _value.size);
             } else {
-                return bf->test_bytes(const_cast<char*>(reinterpret_cast<const char*>(&_value)),
-                                      sizeof(WarpperFieldType));
+                // DecimalV2 using decimal12_t in bloom filter, should convert value to decimal12_t
+                if constexpr (Type == PrimitiveType::TYPE_DECIMALV2) {
+                    decimal12_t decimal12_t_val(_value.int_value(), _value.frac_value());
+                    return bf->test_bytes(

Review Comment:
   what about datev1 and datetimev1, do they also need convert to storage type here? if not please add comment here to explain the reason.



##########
be/src/olap/comparison_predicate.h:
##########
@@ -232,8 +251,16 @@ class ComparisonPredicateBase : public ColumnPredicate {
             if constexpr (std::is_same_v<T, StringRef>) {
                 return bf->test_bytes(_value.data, _value.size);
             } else {
-                return bf->test_bytes(const_cast<char*>(reinterpret_cast<const char*>(&_value)),
-                                      sizeof(WarpperFieldType));
+                // DecimalV2 using decimal12_t in bloom filter, should convert value to decimal12_t
+                if constexpr (Type == PrimitiveType::TYPE_DECIMALV2) {
+                    decimal12_t decimal12_t_val(_value.int_value(), _value.frac_value());
+                    return bf->test_bytes(

Review Comment:
   what about datev1 and datetimev1, do they also need convert to storage type here? if not please add comment here to explain the reason.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](storage) Expressing the types of computation layer and storage layer in PrimitiveTypeTraits [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26191:
URL: https://github.com/apache/doris/pull/26191#issuecomment-1790594944

   TeamCity be ut coverage result:
    Function Coverage: 37.37% (8481/22697) 
    Line Coverage: 29.73% (68589/230698)
    Region Coverage: 28.39% (35496/125045)
    Branch Coverage: 25.12% (18116/72116)
    Coverage Report: http://coverage.selectdb-in.cc/coverage/6cedf0827aba8717ff96a20fab7d4c77d3dc0198_6cedf0827aba8717ff96a20fab7d4c77d3dc0198/report/index.html


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](storage) Expressing the types of computation layer and storage layer in PrimitiveTypeTraits [doris]

Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #26191:
URL: https://github.com/apache/doris/pull/26191#discussion_r1386208584


##########
be/src/vec/columns/column_vector.h:
##########
@@ -198,10 +198,10 @@ class ColumnVector final : public COWHelper<ColumnVectorHelper, ColumnVector<T>>
 
     void insert_date_column(const char* data_ptr, size_t num) {
         data.reserve(data.size() + num);
-        size_t input_value_size = sizeof(uint24_t);
+        constexpr size_t input_value_size = sizeof(uint24_t);
 
         for (int i = 0; i < num; i++) {
-            uint64_t val = 0;
+            uint24_t val = 0;

Review Comment:
   我估计啊,你这么改性能不一定高, 这个24bit 不对齐,可能都不如直接uint32_t



##########
be/src/vec/columns/column_vector.h:
##########
@@ -198,10 +198,10 @@ class ColumnVector final : public COWHelper<ColumnVectorHelper, ColumnVector<T>>
 
     void insert_date_column(const char* data_ptr, size_t num) {
         data.reserve(data.size() + num);
-        size_t input_value_size = sizeof(uint24_t);
+        constexpr size_t input_value_size = sizeof(uint24_t);
 
         for (int i = 0; i < num; i++) {
-            uint64_t val = 0;
+            uint24_t val = 0;

Review Comment:
   我估计啊,你这么改性能不一定高, 这个24bit 不对齐,可能都不如直接uint32_t



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](storage) Expressing the types of computation layer and storage layer in PrimitiveTypeTraits [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26191:
URL: https://github.com/apache/doris/pull/26191#issuecomment-1809318693

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](storage) Expressing the types of computation layer and storage layer in PrimitiveTypeTraits [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26191:
URL: https://github.com/apache/doris/pull/26191#issuecomment-1809421548

   
   <details>
   <summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
   
   ```
   Tpch sf100 test result on commit b8f25b740c1c70e22e69804be79b20b0142f6ae8, data reload: false
   
   run tpch-sf100 query with default conf and session variables
   q1	5282	5199	5170	5170
   q2	364	153	159	153
   q3	2039	2029	1983	1983
   q4	1412	1373	1366	1366
   q5	3958	3966	3999	3966
   q6	258	133	126	126
   q7	1494	875	885	875
   q8	2778	2797	2760	2760
   q9	9569	9415	9453	9415
   q10	3457	3560	3554	3554
   q11	397	261	250	250
   q12	444	301	309	301
   q13	4571	4123	4121	4121
   q14	333	292	290	290
   q15	623	558	539	539
   q16	678	594	582	582
   q17	1122	1072	1073	1072
   q18	8110	7651	7734	7651
   q19	1677	1691	1685	1685
   q20	532	284	296	284
   q21	4706	4387	4394	4387
   q22	513	411	414	411
   Total cold run time: 54317 ms
   Total hot run time: 50941 ms
   
   run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
   q1	5048	5123	5116	5116
   q2	331	227	241	227
   q3	4106	3953	4030	3953
   q4	2848	2760	2782	2760
   q5	9693	9685	9746	9685
   q6	243	123	123	123
   q7	3076	2473	2540	2473
   q8	4852	4863	4901	4863
   q9	13041	13001	12930	12930
   q10	4057	4187	4149	4149
   q11	733	667	672	667
   q12	998	893	852	852
   q13	4328	3885	3908	3885
   q14	368	345	341	341
   q15	595	547	550	547
   q16	744	684	727	684
   q17	3911	3885	3962	3885
   q18	9629	9489	9463	9463
   q19	1844	1778	1757	1757
   q20	2385	2054	2030	2030
   q21	8992	8903	8869	8869
   q22	946	846	874	846
   Total cold run time: 82768 ms
   Total hot run time: 80105 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](storage) Expressing the types of computation layer and storage layer in PrimitiveTypeTraits [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26191:
URL: https://github.com/apache/doris/pull/26191#issuecomment-1807831374

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](storage) Expressing the types of computation layer and storage layer in PrimitiveTypeTraits [doris]

Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #26191:
URL: https://github.com/apache/doris/pull/26191#discussion_r1392564155


##########
be/src/vec/columns/predicate_column.h:
##########
@@ -447,13 +459,9 @@ class PredicateColumnType final : public COWHelper<IColumn, PredicateColumnType<
         // DateV1 and DateTimeV1 is special, its storage format is different from compute format
         // should convert here.
         if constexpr (Type == TYPE_DATE || Type == TYPE_DATETIME) {
-            if constexpr (std::is_same_v<T, uint32_t>) {
-                insert_date_to_res_column(sel, sel_size, column);
-            } else if constexpr (std::is_same_v<T, uint64_t>) {
-                insert_datetime_to_res_column(sel, sel_size, column);
-            } else {
-                LOG(FATAL) << "not reachable";
-            }
+            insert_datetime_to_res_column(sel, sel_size, column);

Review Comment:
   If datev1 using the same type in predicate and compute layer, why do we need use a specific method here?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](storage) Expressing the types of computation layer and storage layer in PrimitiveTypeTraits [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26191:
URL: https://github.com/apache/doris/pull/26191#issuecomment-1812183500

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](storage) Expressing the types of computation layer and storage layer in PrimitiveTypeTraits [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26191:
URL: https://github.com/apache/doris/pull/26191#issuecomment-1790497656

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](storage) Expressing the types of computation layer and storage layer in PrimitiveTypeTraits [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26191:
URL: https://github.com/apache/doris/pull/26191#issuecomment-1790648620

   (From new machine)TeamCity pipeline, clickbench performance test result:
    the sum of best hot time: 46.78 seconds
    stream load tsv:          555 seconds loaded 74807831229 Bytes, about 128 MB/s
    stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
    stream load orc:          65 seconds loaded 1101869774 Bytes, about 16 MB/s
    stream load parquet:          34 seconds loaded 861443392 Bytes, about 24 MB/s
    insert into select:          28.9 seconds inserted 10000000 Rows, about 346K ops/s
    storage size: 17161981252 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](storage) Expressing the types of computation layer and storage layer in PrimitiveTypeTraits [doris]

Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #26191:
URL: https://github.com/apache/doris/pull/26191#issuecomment-1790500292

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](storage) Expressing the types of computation layer and storage layer in PrimitiveTypeTraits [doris]

Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #26191:
URL: https://github.com/apache/doris/pull/26191#discussion_r1393539828


##########
be/src/exprs/bloom_filter_func.h:
##########
@@ -416,37 +416,10 @@ struct FixedStringFindOp : public StringFindOp {
     }
 };
 
-struct DateTimeFindOp : public CommonFindOp<VecDateTimeValue> {
-    bool find_olap_engine(const BloomFilterAdaptor& bloom_filter, const void* data) const {
-        VecDateTimeValue value;
-        value.from_olap_datetime(*reinterpret_cast<const uint64_t*>(data));
-        return bloom_filter.test(Slice((char*)&value, sizeof(VecDateTimeValue)));
-    }
-};
-
-// avoid violating C/C++ aliasing rules.
-// https://gcc.gnu.org/bugzilla/show_bug.cgi?id=101684
-
-struct DateFindOp : public CommonFindOp<VecDateTimeValue> {
-    bool find_olap_engine(const BloomFilterAdaptor& bloom_filter, const void* data) const {
-        uint24_t date = *static_cast<const uint24_t*>(data);
-        uint64_t value = uint32_t(date);
-
-        VecDateTimeValue date_value;
-        date_value.from_olap_date(value);
-
-        return bloom_filter.test(Slice((char*)&date_value, sizeof(VecDateTimeValue)));
-    }
-};
-
 struct DecimalV2FindOp : public CommonFindOp<DecimalV2Value> {
     bool find_olap_engine(const BloomFilterAdaptor& bloom_filter, const void* data) const {
-        auto packed_decimal = *static_cast<const decimal12_t*>(data);
-        DecimalV2Value value;
-        int64_t int_value = packed_decimal.integer;
-        int32_t frac_value = packed_decimal.fraction;
-        value.from_olap_decimal(int_value, frac_value);
-
+        // Predicate column using decimalv2value as column value type, so not need convert here
+        DecimalV2Value value = *static_cast<const DecimalV2Value*>(data);
         constexpr int decimal_value_sz = sizeof(DecimalV2Value);
         char data_bytes[decimal_value_sz];
         memcpy(&data_bytes, &value, decimal_value_sz);

Review Comment:
   remove duplicate memcpy code



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](storage) Expressing the types of computation layer and storage layer in PrimitiveTypeTraits [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26191:
URL: https://github.com/apache/doris/pull/26191#issuecomment-1812238164

   TeamCity be ut coverage result:
    Function Coverage: 36.82% (8423/22879) 
    Line Coverage: 29.29% (68395/233517)
    Region Coverage: 27.90% (35371/126787)
    Branch Coverage: 24.66% (18077/73292)
    Coverage Report: http://coverage.selectdb-in.cc/coverage/b1b0c4a3861d8d4d743abb159ae14f844c585c5d_b1b0c4a3861d8d4d743abb159ae14f844c585c5d/report/index.html


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](storage) Expressing the types of computation layer and storage layer in PrimitiveTypeTraits [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26191:
URL: https://github.com/apache/doris/pull/26191#issuecomment-1809429606

   (From new machine)TeamCity pipeline, clickbench performance test result:
    the sum of best hot time: 47.33 seconds
    stream load tsv:          555 seconds loaded 74807831229 Bytes, about 128 MB/s
    stream load json:         21 seconds loaded 2358488459 Bytes, about 107 MB/s
    stream load orc:          66 seconds loaded 1101869774 Bytes, about 15 MB/s
    stream load parquet:          33 seconds loaded 861443392 Bytes, about 24 MB/s
    insert into select:          28.9 seconds inserted 10000000 Rows, about 346K ops/s
    storage size: 17162099424 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](storage) Expressing the types of computation layer and storage layer in PrimitiveTypeTraits [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26191:
URL: https://github.com/apache/doris/pull/26191#issuecomment-1809346602

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](storage) Expressing the types of computation layer and storage layer in PrimitiveTypeTraits [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26191:
URL: https://github.com/apache/doris/pull/26191#issuecomment-1790594728

   TeamCity be ut coverage result:
    Function Coverage: 37.36% (8481/22698) 
    Line Coverage: 29.73% (68583/230712)
    Region Coverage: 28.38% (35490/125055)
    Branch Coverage: 25.11% (18111/72116)
    Coverage Report: http://coverage.selectdb-in.cc/coverage/cfc940efde290134bcc3a5b74b759683c64a4289_cfc940efde290134bcc3a5b74b759683c64a4289/report/index.html


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](storage) Expressing the types of computation layer and storage layer in PrimitiveTypeTraits [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26191:
URL: https://github.com/apache/doris/pull/26191#issuecomment-1791759364

   (From new machine)TeamCity pipeline, clickbench performance test result:
    the sum of best hot time: 45.92 seconds
    stream load tsv:          555 seconds loaded 74807831229 Bytes, about 128 MB/s
    stream load json:         21 seconds loaded 2358488459 Bytes, about 107 MB/s
    stream load orc:          65 seconds loaded 1101869774 Bytes, about 16 MB/s
    stream load parquet:          32 seconds loaded 861443392 Bytes, about 25 MB/s
    insert into select:          29.1 seconds inserted 10000000 Rows, about 343K ops/s
    storage size: 17162274849 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](storage) Expressing the types of computation layer and storage layer in PrimitiveTypeTraits [doris]

Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #26191:
URL: https://github.com/apache/doris/pull/26191#discussion_r1378871450


##########
be/src/olap/predicate_creator.h:
##########
@@ -49,7 +49,7 @@ class PredicateCreator {
 template <PrimitiveType Type, PredicateType PT, typename ConditionType>
 class IntegerPredicateCreator : public PredicateCreator<ConditionType> {
 public:

Review Comment:
   maybe not right here. because many predicate parse the value from string, and olap scannode convert the predicate  value to a string and predicate creator parse the value from string.
   If you do not change olap scannode code, then I think you could not parse the value correctly.
   I think, maybe we should delete predicate creator.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](storage) Expressing the types of computation layer and storage layer in PrimitiveTypeTraits [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26191:
URL: https://github.com/apache/doris/pull/26191#issuecomment-1800958519

   TeamCity be ut coverage result:
    Function Coverage: 37.02% (8395/22675) 
    Line Coverage: 29.44% (67948/230781)
    Region Coverage: 28.12% (35155/125013)
    Branch Coverage: 24.91% (17965/72126)
    Coverage Report: http://coverage.selectdb-in.cc/coverage/250aad0379597a0309c9d0d87a54a98f92a069b6_250aad0379597a0309c9d0d87a54a98f92a069b6/report/index.html


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](storage) Expressing the types of computation layer and storage layer in PrimitiveTypeTraits [doris]

Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on code in PR #26191:
URL: https://github.com/apache/doris/pull/26191#discussion_r1381926031


##########
be/src/vec/columns/predicate_column.h:
##########
@@ -220,29 +201,60 @@ class PredicateColumnType final : public COWHelper<IColumn, PredicateColumnType<
     }
 
     void insert_many_date(const char* data_ptr, size_t num) {
-        size_t intput_type_size = sizeof(uint24_t);
-        size_t res_type_size = sizeof(uint32_t);
+        constexpr size_t input_type_size = sizeof(PrimitiveTypeTraits<TYPE_DATE>::StorageFieldType);
+        static_assert(input_type_size == sizeof(uint24_t));
         char* input_data_ptr = const_cast<char*>(data_ptr);
 
-        char* res_ptr = (char*)data.get_end_ptr();
-        memset(res_ptr, 0, res_type_size * num);
+        auto* res_ptr = reinterpret_cast<VecDateTimeValue*>(data.get_end_ptr());
         for (int i = 0; i < num; i++) {
-            memcpy(res_ptr, input_data_ptr, intput_type_size);
-            res_ptr += res_type_size;
-            input_data_ptr += intput_type_size;
+            uint64_t value = *reinterpret_cast<uint24_t*>(input_data_ptr);
+            res_ptr[i].set_olap_date(value);
+            input_data_ptr += input_type_size;
+        }
+        data.set_end_ptr(res_ptr + num);
+    }
+
+    void insert_many_datetime(const char* data_ptr, size_t num) {
+        constexpr size_t input_type_size =
+                sizeof(PrimitiveTypeTraits<TYPE_DATETIME>::StorageFieldType);
+        static_assert(input_type_size == sizeof(uint64_t));
+        char* input_data_ptr = const_cast<char*>(data_ptr);
+
+        auto* res_ptr = reinterpret_cast<VecDateTimeValue*>(data.get_end_ptr());
+        for (int i = 0; i < num; i++) {
+            res_ptr[i].from_olap_datetime(*reinterpret_cast<uint64_t*>(input_data_ptr));
+            input_data_ptr += input_type_size;
+        }
+        data.set_end_ptr(res_ptr + num);
+    }
+
+    // The logic is same to ColumnDecimal::insert_many_fix_len_data
+    void insert_many_decimalv2(const char* data_ptr, size_t num) {
+        size_t old_size = data.size();
+        data.resize(old_size + num);
+
+        DecimalV2Value* target = (DecimalV2Value*)(data.data() + old_size);
+        for (int i = 0; i < num; i++) {
+            const char* cur_ptr = data_ptr + sizeof(decimal12_t) * i;
+            int64_t int_value = unaligned_load<int64_t>(cur_ptr);
+            int32_t frac_value = *(int32_t*)(cur_ptr + sizeof(int64_t));
+            target[i].from_olap_decimal(int_value, frac_value);
         }
-        data.set_end_ptr(res_ptr);
     }
 
     void insert_many_fix_len_data(const char* data_ptr, size_t num) override {
-        if constexpr (std::is_same_v<T, decimal12_t>) {

Review Comment:
   For normal primitive types(e.g. TYPE_INT/TYPE_BIGINT/TYPE_DECIMAL32/TYPE_DECIMAL64), there is no need to convert between compute layer type and storage layer type.
   Therefore, we can directly use the `memcpy` function to perform bulk data copying.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](storage) Expressing the types of computation layer and storage layer in PrimitiveTypeTraits [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26191:
URL: https://github.com/apache/doris/pull/26191#issuecomment-1790514572

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](storage) Expressing the types of computation layer and storage layer in PrimitiveTypeTraits [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26191:
URL: https://github.com/apache/doris/pull/26191#issuecomment-1791682207

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](storage) Expressing the types of computation layer and storage layer in PrimitiveTypeTraits [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26191:
URL: https://github.com/apache/doris/pull/26191#issuecomment-1809411968

   (From new machine)TeamCity pipeline, clickbench performance test result:
    the sum of best hot time: 46.5 seconds
    stream load tsv:          554 seconds loaded 74807831229 Bytes, about 128 MB/s
    stream load json:         21 seconds loaded 2358488459 Bytes, about 107 MB/s
    stream load orc:          65 seconds loaded 1101869774 Bytes, about 16 MB/s
    stream load parquet:          32 seconds loaded 861443392 Bytes, about 25 MB/s
    insert into select:          28.8 seconds inserted 10000000 Rows, about 347K ops/s
    storage size: 17162452704 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](storage) Expressing the types of computation layer and storage layer in PrimitiveTypeTraits [doris]

Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #26191:
URL: https://github.com/apache/doris/pull/26191#discussion_r1378863887


##########
be/src/olap/in_list_predicate.h:
##########
@@ -134,9 +121,8 @@ class InListPredicateBase : public ColumnPredicate {
             } else if constexpr (Type == TYPE_DECIMALV2) {
                 HybridSetBase::IteratorBase* iter = hybrid_set->begin();
                 while (iter->has_next()) {
-                    const DecimalV2Value* value = (const DecimalV2Value*)(iter->get_value());
-                    decimal12_t decimal12 = {value->int_value(), value->frac_value()};
-                    _values->insert(&decimal12);
+                    const auto* value = (const DecimalV2Value*)(iter->get_value());
+                    _values->insert(value);
                     iter->next();
                 }
             } else if constexpr (Type == TYPE_DATE) {

Review Comment:
   should also modify datev1 and datetimev1? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](storage) Expressing the types of computation layer and storage layer in PrimitiveTypeTraits [doris]

Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on code in PR #26191:
URL: https://github.com/apache/doris/pull/26191#discussion_r1378875692


##########
be/src/olap/comparison_predicate.h:
##########
@@ -107,8 +120,22 @@ class ComparisonPredicateBase : public ColumnPredicate {
         }
 
         roaring::Roaring roaring;
-        RETURN_IF_ERROR(iterator->read_from_inverted_index(column_name, &_value, query_type,
-                                                           num_rows, &roaring));
+        if constexpr (Type == PrimitiveType::TYPE_DATE) {
+            uint24_t date = _value.to_olap_date();

Review Comment:
   Maybe we can uniformly implement a function for converting CppType and StorageType to each other.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](storage) Expressing the types of computation layer and storage layer in PrimitiveTypeTraits [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26191:
URL: https://github.com/apache/doris/pull/26191#issuecomment-1812317721

   (From new machine)TeamCity pipeline, clickbench performance test result:
    the sum of best hot time: 45.61 seconds
    stream load tsv:          556 seconds loaded 74807831229 Bytes, about 128 MB/s
    stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
    stream load orc:          65 seconds loaded 1101869774 Bytes, about 16 MB/s
    stream load parquet:          32 seconds loaded 861443392 Bytes, about 25 MB/s
    insert into select:          28.3 seconds inserted 10000000 Rows, about 353K ops/s
    storage size: 17097722596 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](storage) Expressing the types of computation layer and storage layer in PrimitiveTypeTraits [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26191:
URL: https://github.com/apache/doris/pull/26191#issuecomment-1808023678

   
   <details>
   <summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
   
   ```
   Tpch sf100 test result on commit f0b58300133758d0bdccf41444ffe7059042ef81, data reload: false
   
   run tpch-sf100 query with default conf and session variables
   q1	5308	5100	5155	5100
   q2	358	161	154	154
   q3	2046	1988	2059	1988
   q4	1414	1373	1362	1362
   q5	3968	3931	4019	3931
   q6	245	129	131	129
   q7	1457	879	907	879
   q8	2790	2790	2779	2779
   q9	9839	9528	9393	9393
   q10	3479	3551	3540	3540
   q11	390	252	257	252
   q12	452	299	300	299
   q13	4560	4129	4148	4129
   q14	320	292	288	288
   q15	653	543	553	543
   q16	693	590	592	590
   q17	1140	1076	1072	1072
   q18	8073	7575	7653	7575
   q19	1686	1684	1709	1684
   q20	579	338	300	300
   q21	4701	4382	4366	4366
   q22	510	423	410	410
   Total cold run time: 54661 ms
   Total hot run time: 50763 ms
   
   run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
   q1	5036	5227	5071	5071
   q2	370	241	263	241
   q3	4073	4076	4082	4076
   q4	2831	2772	2787	2772
   q5	9731	9696	9631	9631
   q6	248	123	123	123
   q7	3077	2506	2526	2506
   q8	4833	4819	4818	4818
   q9	13095	12942	12898	12898
   q10	4108	4202	4203	4202
   q11	810	683	660	660
   q12	1011	851	850	850
   q13	4286	3846	3882	3846
   q14	387	337	360	337
   q15	625	556	547	547
   q16	760	706	683	683
   q17	3909	3999	3947	3947
   q18	9647	9403	9543	9403
   q19	1884	1769	1791	1769
   q20	2422	2089	2065	2065
   q21	9069	9013	8824	8824
   q22	920	845	854	845
   Total cold run time: 83132 ms
   Total hot run time: 80114 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](storage) Expressing the types of computation layer and storage layer in PrimitiveTypeTraits [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26191:
URL: https://github.com/apache/doris/pull/26191#issuecomment-1812476358

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](storage) Expressing the types of computation layer and storage layer in PrimitiveTypeTraits [doris]

Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #26191:
URL: https://github.com/apache/doris/pull/26191#issuecomment-1788309894

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](storage) Expressing the types of computation layer and storage layer in PrimitiveTypeTraits [doris]

Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei merged PR #26191:
URL: https://github.com/apache/doris/pull/26191


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](storage) Expressing the types of computation layer and storage layer in PrimitiveTypeTraits [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26191:
URL: https://github.com/apache/doris/pull/26191#issuecomment-1812355406

   TeamCity be ut coverage result:
    Function Coverage: 36.80% (8418/22875) 
    Line Coverage: 29.29% (68411/233560)
    Region Coverage: 27.88% (35353/126810)
    Branch Coverage: 24.65% (18071/73312)
    Coverage Report: http://coverage.selectdb-in.cc/coverage/6dbc50f5131629ffa647a433564f4c7597afc292_6dbc50f5131629ffa647a433564f4c7597afc292/report/index.html


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](storage) Expressing the types of computation layer and storage layer in PrimitiveTypeTraits [doris]

Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #26191:
URL: https://github.com/apache/doris/pull/26191#issuecomment-1790482597

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](storage) Expressing the types of computation layer and storage layer in PrimitiveTypeTraits [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26191:
URL: https://github.com/apache/doris/pull/26191#issuecomment-1801023714

   (From new machine)TeamCity pipeline, clickbench performance test result:
    the sum of best hot time: 45.14 seconds
    stream load tsv:          556 seconds loaded 74807831229 Bytes, about 128 MB/s
    stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
    stream load orc:          65 seconds loaded 1101869774 Bytes, about 16 MB/s
    stream load parquet:          33 seconds loaded 861443392 Bytes, about 24 MB/s
    insert into select:          29.0 seconds inserted 10000000 Rows, about 344K ops/s
    storage size: 17161946349 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](storage) Expressing the types of computation layer and storage layer in PrimitiveTypeTraits [doris]

Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #26191:
URL: https://github.com/apache/doris/pull/26191#discussion_r1386189902


##########
be/src/exprs/bloom_filter_func.h:
##########
@@ -416,37 +416,10 @@ struct FixedStringFindOp : public StringFindOp {
     }
 };
 
-struct DateTimeFindOp : public CommonFindOp<VecDateTimeValue> {
-    bool find_olap_engine(const BloomFilterAdaptor& bloom_filter, const void* data) const {
-        VecDateTimeValue value;
-        value.from_olap_datetime(*reinterpret_cast<const uint64_t*>(data));
-        return bloom_filter.test(Slice((char*)&value, sizeof(VecDateTimeValue)));
-    }
-};
-
-// avoid violating C/C++ aliasing rules.
-// https://gcc.gnu.org/bugzilla/show_bug.cgi?id=101684
-
-struct DateFindOp : public CommonFindOp<VecDateTimeValue> {
-    bool find_olap_engine(const BloomFilterAdaptor& bloom_filter, const void* data) const {
-        uint24_t date = *static_cast<const uint24_t*>(data);
-        uint64_t value = uint32_t(date);
-
-        VecDateTimeValue date_value;
-        date_value.from_olap_date(value);
-
-        return bloom_filter.test(Slice((char*)&date_value, sizeof(VecDateTimeValue)));
-    }
-};
-
 struct DecimalV2FindOp : public CommonFindOp<DecimalV2Value> {
     bool find_olap_engine(const BloomFilterAdaptor& bloom_filter, const void* data) const {
-        auto packed_decimal = *static_cast<const decimal12_t*>(data);
-        DecimalV2Value value;
-        int64_t int_value = packed_decimal.integer;
-        int32_t frac_value = packed_decimal.fraction;
-        value.from_olap_decimal(int_value, frac_value);
-
+        // Predicate column using decimalv2value as column value type, so not need convert here
+        DecimalV2Value value = *static_cast<const DecimalV2Value*>(data);

Review Comment:
   Could we delete DecimalV2FindOp??



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](storage) Expressing the types of computation layer and storage layer in PrimitiveTypeTraits [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26191:
URL: https://github.com/apache/doris/pull/26191#issuecomment-1791729348

   TeamCity be ut coverage result:
    Function Coverage: 37.28% (8453/22673) 
    Line Coverage: 29.66% (68405/230619)
    Region Coverage: 28.34% (35414/124983)
    Branch Coverage: 25.07% (18069/72076)
    Coverage Report: http://coverage.selectdb-in.cc/coverage/741fb6fde2b24deb28ad8fe75457b1c7410e2147_741fb6fde2b24deb28ad8fe75457b1c7410e2147/report/index.html


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](storage) Expressing the types of computation layer and storage layer in PrimitiveTypeTraits [doris]

Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on code in PR #26191:
URL: https://github.com/apache/doris/pull/26191#discussion_r1381320218


##########
be/src/vec/columns/predicate_column.h:
##########
@@ -220,29 +201,60 @@ class PredicateColumnType final : public COWHelper<IColumn, PredicateColumnType<
     }
 
     void insert_many_date(const char* data_ptr, size_t num) {
-        size_t intput_type_size = sizeof(uint24_t);
-        size_t res_type_size = sizeof(uint32_t);
+        constexpr size_t input_type_size = sizeof(PrimitiveTypeTraits<TYPE_DATE>::StorageFieldType);
+        static_assert(input_type_size == sizeof(uint24_t));
         char* input_data_ptr = const_cast<char*>(data_ptr);
 
-        char* res_ptr = (char*)data.get_end_ptr();
-        memset(res_ptr, 0, res_type_size * num);
+        auto* res_ptr = reinterpret_cast<VecDateTimeValue*>(data.get_end_ptr());
         for (int i = 0; i < num; i++) {
-            memcpy(res_ptr, input_data_ptr, intput_type_size);
-            res_ptr += res_type_size;
-            input_data_ptr += intput_type_size;
+            uint64_t value = *reinterpret_cast<uint24_t*>(input_data_ptr);
+            res_ptr[i].set_olap_date(value);
+            input_data_ptr += input_type_size;
+        }
+        data.set_end_ptr(res_ptr + num);
+    }
+
+    void insert_many_datetime(const char* data_ptr, size_t num) {
+        constexpr size_t input_type_size =
+                sizeof(PrimitiveTypeTraits<TYPE_DATETIME>::StorageFieldType);
+        static_assert(input_type_size == sizeof(uint64_t));
+        char* input_data_ptr = const_cast<char*>(data_ptr);
+
+        auto* res_ptr = reinterpret_cast<VecDateTimeValue*>(data.get_end_ptr());
+        for (int i = 0; i < num; i++) {
+            res_ptr[i].from_olap_datetime(*reinterpret_cast<uint64_t*>(input_data_ptr));
+            input_data_ptr += input_type_size;
+        }
+        data.set_end_ptr(res_ptr + num);
+    }
+
+    // The logic is same to ColumnDecimal::insert_many_fix_len_data
+    void insert_many_decimalv2(const char* data_ptr, size_t num) {
+        size_t old_size = data.size();
+        data.resize(old_size + num);
+
+        DecimalV2Value* target = (DecimalV2Value*)(data.data() + old_size);
+        for (int i = 0; i < num; i++) {
+            const char* cur_ptr = data_ptr + sizeof(decimal12_t) * i;
+            int64_t int_value = unaligned_load<int64_t>(cur_ptr);
+            int32_t frac_value = *(int32_t*)(cur_ptr + sizeof(int64_t));
+            target[i].from_olap_decimal(int_value, frac_value);
         }
-        data.set_end_ptr(res_ptr);
     }
 
     void insert_many_fix_len_data(const char* data_ptr, size_t num) override {
-        if constexpr (std::is_same_v<T, decimal12_t>) {

Review Comment:
   Add a method to convert from StorageFieldType to CppType, and then we can remove the if else in get_zone_map_value/insert_many_fix_len_data, and all methods like insert_many_date/insert_many_decimalv2...



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](storage) Expressing the types of computation layer and storage layer in PrimitiveTypeTraits [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26191:
URL: https://github.com/apache/doris/pull/26191#issuecomment-1812425740

   
   <details>
   <summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
   
   ```
   Tpch sf100 test result on commit 6dbc50f5131629ffa647a433564f4c7597afc292, data reload: false
   
   run tpch-sf100 query with default conf and session variables
   q1	5246	5039	5029	5029
   q2	366	156	160	156
   q3	2029	1921	1921	1921
   q4	1375	1265	1244	1244
   q5	3930	3900	3987	3900
   q6	250	133	131	131
   q7	1407	883	871	871
   q8	2725	2787	2757	2757
   q9	13738	9946	9614	9614
   q10	3441	3519	3519	3519
   q11	396	258	248	248
   q12	447	294	290	290
   q13	4599	3827	3776	3776
   q14	318	288	277	277
   q15	628	564	566	564
   q16	655	585	580	580
   q17	1119	954	899	899
   q18	7880	7409	7354	7354
   q19	1684	1678	1672	1672
   q20	574	297	294	294
   q21	4449	3949	4024	3949
   q22	517	417	404	404
   Total cold run time: 57773 ms
   Total hot run time: 49449 ms
   
   run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
   q1	5051	5093	5008	5008
   q2	326	221	225	221
   q3	4061	3865	3866	3865
   q4	2798	2755	2742	2742
   q5	9562	9532	9473	9473
   q6	243	127	128	127
   q7	2682	2281	2329	2281
   q8	4852	4861	4837	4837
   q9	13261	13058	13101	13058
   q10	4061	4175	4166	4166
   q11	754	657	660	657
   q12	1008	838	828	828
   q13	4276	3654	3561	3561
   q14	366	348	345	345
   q15	580	563	550	550
   q16	755	657	695	657
   q17	3855	3862	3949	3862
   q18	9457	8901	8996	8901
   q19	1819	1755	1785	1755
   q20	2406	2048	2038	2038
   q21	8706	8460	8601	8460
   q22	928	802	853	802
   Total cold run time: 81807 ms
   Total hot run time: 78194 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](storage) Expressing the types of computation layer and storage layer in PrimitiveTypeTraits [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26191:
URL: https://github.com/apache/doris/pull/26191#issuecomment-1812436602

   (From new machine)TeamCity pipeline, clickbench performance test result:
    the sum of best hot time: 45.55 seconds
    stream load tsv:          582 seconds loaded 74807831229 Bytes, about 122 MB/s
    stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
    stream load orc:          65 seconds loaded 1101869774 Bytes, about 16 MB/s
    stream load parquet:          32 seconds loaded 861443392 Bytes, about 25 MB/s
    insert into select:          29.0 seconds inserted 10000000 Rows, about 344K ops/s
    storage size: 17098197739 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](storage) Expressing the types of computation layer and storage layer in PrimitiveTypeTraits [doris]

Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #26191:
URL: https://github.com/apache/doris/pull/26191#issuecomment-1800870179

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](storage) Expressing the types of computation layer and storage layer in PrimitiveTypeTraits [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26191:
URL: https://github.com/apache/doris/pull/26191#issuecomment-1808018311

   (From new machine)TeamCity pipeline, clickbench performance test result:
    the sum of best hot time: 44.84 seconds
    stream load tsv:          553 seconds loaded 74807831229 Bytes, about 129 MB/s
    stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
    stream load orc:          65 seconds loaded 1101869774 Bytes, about 16 MB/s
    stream load parquet:          32 seconds loaded 861443392 Bytes, about 25 MB/s
    insert into select:          29.0 seconds inserted 10000000 Rows, about 344K ops/s
    storage size: 17162250011 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](storage) Expressing the types of computation layer and storage layer in PrimitiveTypeTraits [doris]

Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #26191:
URL: https://github.com/apache/doris/pull/26191#discussion_r1392558588


##########
be/src/olap/in_list_predicate.h:
##########
@@ -363,16 +319,16 @@ class InListPredicateBase : public ColumnPredicate {
         }
         if constexpr (PT == PredicateType::NOT_IN_LIST) {
             if constexpr (Type == TYPE_DATE) {
-                T tmp_min_uint32_value = 0;

Review Comment:
   if constexpr (PT == PredicateType::IN_LIST) {
   
   why not using the same way?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](storage) Expressing the types of computation layer and storage layer in PrimitiveTypeTraits [doris]

Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #26191:
URL: https://github.com/apache/doris/pull/26191#issuecomment-1812085521

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](storage) Expressing the types of computation layer and storage layer in PrimitiveTypeTraits [doris]

Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #26191:
URL: https://github.com/apache/doris/pull/26191#issuecomment-1812169591

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](storage) Expressing the types of computation layer and storage layer in PrimitiveTypeTraits [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26191:
URL: https://github.com/apache/doris/pull/26191#issuecomment-1812185784

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](storage) Expressing the types of computation layer and storage layer in PrimitiveTypeTraits [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26191:
URL: https://github.com/apache/doris/pull/26191#issuecomment-1812004440

   TeamCity be ut coverage result:
    Function Coverage: 36.80% (8420/22879) 
    Line Coverage: 29.29% (68403/233518)
    Region Coverage: 27.88% (35346/126786)
    Branch Coverage: 24.64% (18062/73292)
    Coverage Report: http://coverage.selectdb-in.cc/coverage/36a7b5f5ad50854548e52cc4b35d34bc0e08edc8_36a7b5f5ad50854548e52cc4b35d34bc0e08edc8/report/index.html


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](storage) Expressing the types of computation layer and storage layer in PrimitiveTypeTraits [doris]

Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #26191:
URL: https://github.com/apache/doris/pull/26191#issuecomment-1800891263

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](storage) Expressing the types of computation layer and storage layer in PrimitiveTypeTraits [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26191:
URL: https://github.com/apache/doris/pull/26191#issuecomment-1800879306

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](storage) Expressing the types of computation layer and storage layer in PrimitiveTypeTraits [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26191:
URL: https://github.com/apache/doris/pull/26191#issuecomment-1803025956

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](storage) Expressing the types of computation layer and storage layer in PrimitiveTypeTraits [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26191:
URL: https://github.com/apache/doris/pull/26191#issuecomment-1809403600

   TeamCity be ut coverage result:
    Function Coverage: 36.75% (8407/22875) 
    Line Coverage: 29.29% (68152/232704)
    Region Coverage: 27.85% (35182/126325)
    Branch Coverage: 24.66% (17974/72894)
    Coverage Report: http://coverage.selectdb-in.cc/coverage/b8f25b740c1c70e22e69804be79b20b0142f6ae8_b8f25b740c1c70e22e69804be79b20b0142f6ae8/report/index.html


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](storage) Expressing the types of computation layer and storage layer in PrimitiveTypeTraits [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26191:
URL: https://github.com/apache/doris/pull/26191#issuecomment-1807955280

   TeamCity be ut coverage result:
    Function Coverage: 36.76% (8408/22873) 
    Line Coverage: 29.30% (68163/232654)
    Region Coverage: 27.86% (35182/126298)
    Branch Coverage: 24.67% (17978/72868)
    Coverage Report: http://coverage.selectdb-in.cc/coverage/f0b58300133758d0bdccf41444ffe7059042ef81_f0b58300133758d0bdccf41444ffe7059042ef81/report/index.html


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](storage) Expressing the types of computation layer and storage layer in PrimitiveTypeTraits [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26191:
URL: https://github.com/apache/doris/pull/26191#issuecomment-1809382815

   TeamCity be ut coverage result:
    Function Coverage: 36.75% (8406/22875) 
    Line Coverage: 29.28% (68142/232694)
    Region Coverage: 27.85% (35177/126316)
    Branch Coverage: 24.66% (17976/72890)
    Coverage Report: http://coverage.selectdb-in.cc/coverage/aac0208bf8be26d516eaaecd2d657012e59a2a68_aac0208bf8be26d516eaaecd2d657012e59a2a68/report/index.html


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](storage) Expressing the types of computation layer and storage layer in PrimitiveTypeTraits [doris]

Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #26191:
URL: https://github.com/apache/doris/pull/26191#issuecomment-1807806077

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](storage) Expressing the types of computation layer and storage layer in PrimitiveTypeTraits [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26191:
URL: https://github.com/apache/doris/pull/26191#issuecomment-1788559952

   TeamCity be ut coverage result:
    Function Coverage: 37.34% (8470/22682) 
    Line Coverage: 29.66% (68561/231177)
    Region Coverage: 28.23% (35608/126152)
    Branch Coverage: 25.09% (18224/72622)
    Coverage Report: http://coverage.selectdb-in.cc/coverage/f088cab0b831ec8556731e9e920d468d381d00a7_f088cab0b831ec8556731e9e920d468d381d00a7/report/index.html


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](storage) Expressing the types of computation layer and storage layer in PrimitiveTypeTraits [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26191:
URL: https://github.com/apache/doris/pull/26191#issuecomment-1790647437

   (From new machine)TeamCity pipeline, clickbench performance test result:
    the sum of best hot time: 46.57 seconds
    stream load tsv:          553 seconds loaded 74807831229 Bytes, about 129 MB/s
    stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
    stream load orc:          65 seconds loaded 1101869774 Bytes, about 16 MB/s
    stream load parquet:          33 seconds loaded 861443392 Bytes, about 24 MB/s
    insert into select:          28.9 seconds inserted 10000000 Rows, about 346K ops/s
    storage size: 17161970149 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org