You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2022/04/12 06:47:49 UTC

[GitHub] [incubator-doris] wangbo opened a new pull request, #8967: [optimize](storage)optimize date in storage layer

wangbo opened a new pull request, #8967:
URL: https://github.com/apache/incubator-doris/pull/8967

   # Proposed changes
   
   Issue Number: close #8796
   
   ## Problem Summary:
   
   Describe the overview of changes.
   
   ## Checklist(Required)
   
   1. Does it affect the original behavior: (Yes/No/I Don't know)
   2. Has unit tests been added: (Yes/No/No Need)
   3. Has document been added or modified: (Yes/No/No Need)
   4. Does it need to update dependencies: (Yes/No)
   5. Are there any changes that cannot be rolled back: (Yes/No)
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [incubator-doris] wangbo commented on pull request #8967: [optimize](storage)optimize date in storage layer

Posted by GitBox <gi...@apache.org>.
wangbo commented on PR #8967:
URL: https://github.com/apache/incubator-doris/pull/8967#issuecomment-1096193037

   #  Performance Test
   ## SQL1
   ```
   SELECT sum(LO_EXTENDEDPRICE * LO_DISCOUNT) AS revenue
   FROM lineorder_flat
   WHERE LO_ORDERDATE >= 19930101 and LO_ORDERDATE <= 19931231 AND LO_DISCOUNT BETWEEN 1 AND 3 AND LO_QUANTITY < 25;
   
   before:
   - BlockLoadTime: 1s226ms
   - PredColumnReadTime: 734.907ms
   - VectorPredEvalTime: 296.752ms
   - OutputColumnTime: 179.064ms
   
   after:
   - BlockLoadTime: 903.229ms
   - PredColumnReadTime: 738.852ms
   - VectorPredEvalTime: 83.223ms
   - OutputColumnTime: 65.080ms
   ```
   
   ## SQL2:  pred only contains date
   ```
   SELECT sum(LO_DISCOUNT) AS revenue
   FROM lineorder_flat
   WHERE LO_ORDERDATE >= 19930101 and LO_ORDERDATE <= 19931231
   
   before:
   - BlockLoadTime: 1s982ms
   - PredColumnReadTime: 478.911ms
   - VectorPredEvalTime: 307.892ms
   - OutputColumnTime: 1s180ms
   
   after:
   - BlockLoadTime: 879.068ms
   - PredColumnReadTime: 489.480ms
   - VectorPredEvalTime: 100.503ms
   - OutputColumnTime: 274.997ms
   
   ```
   
   ## SQL3: no predicate
   ```
   select count(distinct LO_ORDERDATE) from lineorder_flat
   
   before:
   - BlockLoadTime: 8s395ms
   - PredColumnReadTime: 8s002ms
   
   after:
   - BlockLoadTime: 6s641ms
   - PredColumnReadTime: 6s252ms
   ```
   
   We can see ```VectorPredEvalTime```  has significant performance improvement.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [incubator-doris] wangbo commented on a diff in pull request #8967: [optimize](storage)optimize date in storage layer

Posted by GitBox <gi...@apache.org>.
wangbo commented on code in PR #8967:
URL: https://github.com/apache/incubator-doris/pull/8967#discussion_r848353898


##########
be/src/vec/columns/column_vector.h:
##########
@@ -168,18 +168,16 @@ class ColumnVector final : public COWHelper<ColumnVectorHelper, ColumnVector<T>>
     }
 
     void insert_date_column(const char* data_ptr, size_t num) {
-        size_t value_size = sizeof(uint24_t);
+        size_t input_value_size = sizeof(uint24_t);
+        
         for (int i = 0; i < num; i++) {
-            const char* cur_ptr = data_ptr + value_size * i;
-            uint64_t value = 0;
-            value = *(unsigned char*)(cur_ptr + 2);
-            value <<= 8;
-            value |= *(unsigned char*)(cur_ptr + 1);
-            value <<= 8;
-            value |= *(unsigned char*)(cur_ptr);
-            vectorized::VecDateTimeValue date;
-            date.from_olap_date(value);
-            this->insert_data(reinterpret_cast<char*>(&date), 0);
+            uint64_t val = 0;
+            memcpy((char*)(&val), data_ptr, input_value_size);
+            data_ptr += input_value_size;
+
+            VecDateTimeValue date;

Review Comment:
   ```
   q1
   - OutputColumnTime: 90.720ms
   q2
   - OutputColumnTime: 286.477ms
   q3
   - PredColumnReadTime: 6s827ms
   ```
   I did test after move it out of loop.
   There is no noticeable performance change.
   I guess  the cost of allocate a Object like ```VecDateTimeValue``` on stack  is similar to allocate a int64.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [incubator-doris] wangbo commented on a diff in pull request #8967: [optimize](storage)optimize date in storage layer

Posted by GitBox <gi...@apache.org>.
wangbo commented on code in PR #8967:
URL: https://github.com/apache/incubator-doris/pull/8967#discussion_r898619934


##########
be/src/vec/columns/predicate_column.h:
##########
@@ -405,6 +435,9 @@ class PredicateColumnType final : public COWHelper<IColumn, PredicateColumnType<
         } else if constexpr (std::is_same_v<T, uint24_t>) {

Review Comment:
   No, compile could failed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [incubator-doris] zenoyang commented on a diff in pull request #8967: [optimize](storage)optimize date in storage layer

Posted by GitBox <gi...@apache.org>.
zenoyang commented on code in PR #8967:
URL: https://github.com/apache/incubator-doris/pull/8967#discussion_r848388700


##########
be/src/vec/columns/predicate_column.h:
##########
@@ -205,13 +218,30 @@ class PredicateColumnType final : public COWHelper<IColumn, PredicateColumnType<
         }
     }
 
+    void insert_many_date(const char* data_ptr, size_t num) {
+        size_t intput_type_size  = sizeof(uint24_t);
+        size_t res_type_size = sizeof(uint32_t);
+        char* input_data_ptr = const_cast<char*>(data_ptr);
+        
+        char* res_ptr = (char*)data.get_end_ptr();
+        memset(res_ptr, 0, res_type_size * num);
+        for (int i = 0; i < num; i++) {

Review Comment:
   ok



##########
be/src/vec/columns/column_vector.h:
##########
@@ -168,18 +168,16 @@ class ColumnVector final : public COWHelper<ColumnVectorHelper, ColumnVector<T>>
     }
 
     void insert_date_column(const char* data_ptr, size_t num) {
-        size_t value_size = sizeof(uint24_t);
+        size_t input_value_size = sizeof(uint24_t);
+        
         for (int i = 0; i < num; i++) {
-            const char* cur_ptr = data_ptr + value_size * i;
-            uint64_t value = 0;
-            value = *(unsigned char*)(cur_ptr + 2);
-            value <<= 8;
-            value |= *(unsigned char*)(cur_ptr + 1);
-            value <<= 8;
-            value |= *(unsigned char*)(cur_ptr);
-            vectorized::VecDateTimeValue date;
-            date.from_olap_date(value);
-            this->insert_data(reinterpret_cast<char*>(&date), 0);
+            uint64_t val = 0;
+            memcpy((char*)(&val), data_ptr, input_value_size);
+            data_ptr += input_value_size;
+
+            VecDateTimeValue date;

Review Comment:
   ok



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [incubator-doris] zenoyang commented on a diff in pull request #8967: [optimize](storage)optimize date in storage layer

Posted by GitBox <gi...@apache.org>.
zenoyang commented on code in PR #8967:
URL: https://github.com/apache/incubator-doris/pull/8967#discussion_r848265385


##########
be/src/vec/columns/predicate_column.h:
##########
@@ -205,13 +218,30 @@ class PredicateColumnType final : public COWHelper<IColumn, PredicateColumnType<
         }
     }
 
+    void insert_many_date(const char* data_ptr, size_t num) {
+        size_t intput_type_size  = sizeof(uint24_t);
+        size_t res_type_size = sizeof(uint32_t);
+        char* input_data_ptr = const_cast<char*>(data_ptr);
+        
+        char* res_ptr = (char*)data.get_end_ptr();
+        memset(res_ptr, 0, res_type_size * num);
+        for (int i = 0; i < num; i++) {

Review Comment:
   Why not batch memcpy?



##########
be/src/vec/runtime/vdatetime_value.h:
##########
@@ -203,6 +203,21 @@ class VecDateTimeValue { // Now this type is a temp solution with little changes
         return check_range_and_set_time(year, month, day, hour, minute, second, _type);
     }
 
+    //note(wb) not check in this method
+    void inline set_olap_date(uint64_t olap_date_val) {
+        _neg = 0;
+        _type = TIME_DATE;
+
+         _day = olap_date_val & 0x1f;
+        _month = (olap_date_val >> 5) & 0x0f;
+        _year = olap_date_val >> 9;
+        _neg = 0;
+        _type = TIME_DATE;

Review Comment:
   ```
   _neg = 0;
   _type = TIME_DATE;
   ```
   Duplicated with the code above



##########
be/src/vec/columns/column_vector.h:
##########
@@ -168,18 +168,16 @@ class ColumnVector final : public COWHelper<ColumnVectorHelper, ColumnVector<T>>
     }
 
     void insert_date_column(const char* data_ptr, size_t num) {
-        size_t value_size = sizeof(uint24_t);
+        size_t input_value_size = sizeof(uint24_t);
+        
         for (int i = 0; i < num; i++) {
-            const char* cur_ptr = data_ptr + value_size * i;
-            uint64_t value = 0;
-            value = *(unsigned char*)(cur_ptr + 2);
-            value <<= 8;
-            value |= *(unsigned char*)(cur_ptr + 1);
-            value <<= 8;
-            value |= *(unsigned char*)(cur_ptr);
-            vectorized::VecDateTimeValue date;
-            date.from_olap_date(value);
-            this->insert_data(reinterpret_cast<char*>(&date), 0);
+            uint64_t val = 0;
+            memcpy((char*)(&val), data_ptr, input_value_size);
+            data_ptr += input_value_size;
+
+            VecDateTimeValue date;

Review Comment:
   ```VecDateTimeValue date;```
   Better to move outside the loop.



##########
be/src/vec/columns/predicate_column.h:
##########
@@ -64,6 +64,19 @@ class PredicateColumnType final : public COWHelper<IColumn, PredicateColumnType<
         }
     }
 
+    void insert_date32_to_res_column(const uint16_t* sel, size_t sel_size,
+                                   vectorized::ColumnVector<Int64>* res_ptr) {
+        res_ptr->reserve(sel_size);
+        auto& res_data = res_ptr->get_data();
+
+        for (size_t i = 0; i < sel_size; i++) {
+            VecDateTimeValue date;

Review Comment:
   same as above



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [doris] yiguolei merged pull request #8967: [optimize](storage)optimize date in storage layer

Posted by GitBox <gi...@apache.org>.
yiguolei merged PR #8967:
URL: https://github.com/apache/doris/pull/8967


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [incubator-doris] wangbo commented on a diff in pull request #8967: [optimize](storage)optimize date in storage layer

Posted by GitBox <gi...@apache.org>.
wangbo commented on code in PR #8967:
URL: https://github.com/apache/incubator-doris/pull/8967#discussion_r848334514


##########
be/src/vec/columns/predicate_column.h:
##########
@@ -205,13 +218,30 @@ class PredicateColumnType final : public COWHelper<IColumn, PredicateColumnType<
         }
     }
 
+    void insert_many_date(const char* data_ptr, size_t num) {
+        size_t intput_type_size  = sizeof(uint24_t);
+        size_t res_type_size = sizeof(uint32_t);
+        char* input_data_ptr = const_cast<char*>(data_ptr);
+        
+        char* res_ptr = (char*)data.get_end_ptr();
+        memset(res_ptr, 0, res_type_size * num);
+        for (int i = 0; i < num; i++) {

Review Comment:
   ```input_data_ptr``` is a uint24_t array.
   ```res_ptr``` is a uint32_t array.
   The element length is different.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [incubator-doris] wangbo commented on a diff in pull request #8967: [optimize](storage)optimize date in storage layer

Posted by GitBox <gi...@apache.org>.
wangbo commented on code in PR #8967:
URL: https://github.com/apache/incubator-doris/pull/8967#discussion_r848353898


##########
be/src/vec/columns/column_vector.h:
##########
@@ -168,18 +168,16 @@ class ColumnVector final : public COWHelper<ColumnVectorHelper, ColumnVector<T>>
     }
 
     void insert_date_column(const char* data_ptr, size_t num) {
-        size_t value_size = sizeof(uint24_t);
+        size_t input_value_size = sizeof(uint24_t);
+        
         for (int i = 0; i < num; i++) {
-            const char* cur_ptr = data_ptr + value_size * i;
-            uint64_t value = 0;
-            value = *(unsigned char*)(cur_ptr + 2);
-            value <<= 8;
-            value |= *(unsigned char*)(cur_ptr + 1);
-            value <<= 8;
-            value |= *(unsigned char*)(cur_ptr);
-            vectorized::VecDateTimeValue date;
-            date.from_olap_date(value);
-            this->insert_data(reinterpret_cast<char*>(&date), 0);
+            uint64_t val = 0;
+            memcpy((char*)(&val), data_ptr, input_value_size);
+            data_ptr += input_value_size;
+
+            VecDateTimeValue date;

Review Comment:
   ```
   q1
   - OutputColumnTime: 90.720ms
   q2
   - OutputColumnTime: 286.477ms
   q3
   - PredColumnReadTime: 6s827ms
   ```
   I did test after move it out of loop.
   There is no noticeable performance change.
   I guess  the cost of allocate a Object like ```VecDateTimeValue```  is similar to allocate a int64.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [incubator-doris] Gabriel39 commented on a diff in pull request #8967: [optimize](storage)optimize date in storage layer

Posted by GitBox <gi...@apache.org>.
Gabriel39 commented on code in PR #8967:
URL: https://github.com/apache/incubator-doris/pull/8967#discussion_r897762753


##########
be/src/vec/columns/predicate_column.h:
##########
@@ -405,6 +435,9 @@ class PredicateColumnType final : public COWHelper<IColumn, PredicateColumnType<
         } else if constexpr (std::is_same_v<T, uint24_t>) {

Review Comment:
   could this code path be deleted?



##########
be/src/vec/runtime/vdatetime_value.h:
##########
@@ -223,6 +223,19 @@ class VecDateTimeValue { // Now this type is a temp solution with little changes
         return check_range_and_set_time(year, month, day, hour, minute, second, _type);
     }
 
+    //note(wb) not check in this method

Review Comment:
   why not use `from_olap_date` above instead?



##########
be/src/vec/runtime/vdatetime_value.h:
##########
@@ -223,6 +223,19 @@ class VecDateTimeValue { // Now this type is a temp solution with little changes
         return check_range_and_set_time(year, month, day, hour, minute, second, _type);
     }
 
+    //note(wb) not check in this method

Review Comment:
   And I'm not sure if it's reasonable to not check range here. Any ideas here?



##########
be/src/vec/columns/column_vector.h:
##########
@@ -169,17 +169,16 @@ class ColumnVector final : public COWHelper<ColumnVectorHelper, ColumnVector<T>>
     }
 
     void insert_date_column(const char* data_ptr, size_t num) {
-        size_t value_size = sizeof(uint24_t);
+        size_t input_value_size = sizeof(uint24_t);
+        
         for (int i = 0; i < num; i++) {

Review Comment:
   better to use pointer directly instead of `i`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [doris] github-actions[bot] commented on pull request #8967: [optimize](storage)optimize date in storage layer

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #8967:
URL: https://github.com/apache/doris/pull/8967#issuecomment-1163900156

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [doris] github-actions[bot] commented on pull request #8967: [optimize](storage)optimize date in storage layer

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #8967:
URL: https://github.com/apache/doris/pull/8967#issuecomment-1163900122

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [doris] wangbo commented on a diff in pull request #8967: [optimize](storage)optimize date in storage layer

Posted by GitBox <gi...@apache.org>.
wangbo commented on code in PR #8967:
URL: https://github.com/apache/doris/pull/8967#discussion_r902130781


##########
be/src/vec/columns/column_vector.h:
##########
@@ -169,17 +169,16 @@ class ColumnVector final : public COWHelper<ColumnVectorHelper, ColumnVector<T>>
     }
 
     void insert_date_column(const char* data_ptr, size_t num) {
-        size_t value_size = sizeof(uint24_t);
+        size_t input_value_size = sizeof(uint24_t);
+        
         for (int i = 0; i < num; i++) {

Review Comment:
   What's the difference?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [incubator-doris] wangbo commented on a diff in pull request #8967: [optimize](storage)optimize date in storage layer

Posted by GitBox <gi...@apache.org>.
wangbo commented on code in PR #8967:
URL: https://github.com/apache/incubator-doris/pull/8967#discussion_r898620159


##########
be/src/vec/runtime/vdatetime_value.h:
##########
@@ -223,6 +223,19 @@ class VecDateTimeValue { // Now this type is a temp solution with little changes
         return check_range_and_set_time(year, month, day, hour, minute, second, _type);
     }
 
+    //note(wb) not check in this method

Review Comment:
   Only used for read from storage layer is ok.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org