You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "mrhhsg (via GitHub)" <gi...@apache.org> on 2023/08/18 07:49:44 UTC

[GitHub] [doris] mrhhsg opened a new pull request, #23172: [improvement](agg) Do not serialize bitmap to string

mrhhsg opened a new pull request, #23172:
URL: https://github.com/apache/doris/pull/23172

   ## Proposed changes
   
   There is no need to serialize bitmap into a string because bitmap can be serialized into protobuf.
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [doris] yiguolei commented on a diff in pull request #23172: [improvement](agg) Do not serialize bitmap to string

Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #23172:
URL: https://github.com/apache/doris/pull/23172#discussion_r1299128167


##########
fe/fe-common/src/main/java/org/apache/doris/common/Config.java:
##########
@@ -1619,7 +1619,7 @@ public class Config extends ConfigBase {
      * Max data version of backends serialize block.
      */
     @ConfField(mutable = false)
-    public static int max_be_exec_version = 2;
+    public static int max_be_exec_version = 3;

Review Comment:
   Add comment here, why need change be exec version to 3??



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [doris] HappenLee commented on a diff in pull request #23172: [improvement](agg) Do not serialize bitmap to string

Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on code in PR #23172:
URL: https://github.com/apache/doris/pull/23172#discussion_r1299145114


##########
be/src/vec/aggregate_functions/aggregate_function_bitmap.h:
##########
@@ -146,10 +146,127 @@ struct AggregateFunctionBitmapData {
     BitmapValue& get() { return value; }
 };
 
+template <typename Data, typename Derived>
+class AggregateFunctionBitmapSerializationHelper
+        : public IAggregateFunctionDataHelper<Data, Derived> {
+public:
+    using BaseHelper = IAggregateFunctionHelper<Derived>;
+
+    AggregateFunctionBitmapSerializationHelper(const DataTypes& argument_types_)
+            : IAggregateFunctionDataHelper<Data, Derived>(argument_types_) {}
+
+    void serialize_to_column(const std::vector<AggregateDataPtr>& places, size_t offset,
+                             MutableColumnPtr& dst, const size_t num_rows) const override {
+        if (version >= 3) {
+            auto& col = assert_cast<ColumnBitmap&>(*dst);
+            col.resize(num_rows);
+            auto* data = col.get_data().data();
+            for (size_t i = 0; i != num_rows; ++i) {
+                data[i] = this->data(places[i] + offset).value;

Review Comment:
   better use `std::move(this->data(places[i] + offset).value)`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [doris] mrhhsg commented on pull request #23172: [improvement](agg) Do not serialize bitmap to string

Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #23172:
URL: https://github.com/apache/doris/pull/23172#issuecomment-1683510007

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [doris] github-actions[bot] commented on pull request #23172: [improvement](agg) Do not serialize bitmap to string

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #23172:
URL: https://github.com/apache/doris/pull/23172#issuecomment-1684856804

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [doris] hello-stephen commented on pull request #23172: [improvement](agg) Do not serialize bitmap to string

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #23172:
URL: https://github.com/apache/doris/pull/23172#issuecomment-1684017375

   (From new machine)TeamCity pipeline, clickbench performance test result:
    the sum of best hot time: 45.2 seconds
    stream load tsv:          545 seconds loaded 74807831229 Bytes, about 130 MB/s
    stream load json:         21 seconds loaded 2358488459 Bytes, about 107 MB/s
    stream load orc:          65 seconds loaded 1101869774 Bytes, about 16 MB/s
    stream load parquet:          31 seconds loaded 861443392 Bytes, about 26 MB/s
    insert into select:          29.3 seconds inserted 10000000 Rows, about 341K ops/s
    storage size: 17162263326 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [doris] hello-stephen commented on pull request #23172: [improvement](agg) Do not serialize bitmap to string

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #23172:
URL: https://github.com/apache/doris/pull/23172#issuecomment-1683592897

   (From new machine)TeamCity pipeline, clickbench performance test result:
    the sum of best hot time: 48.11 seconds
    stream load tsv:          543 seconds loaded 74807831229 Bytes, about 131 MB/s
    stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
    stream load orc:          65 seconds loaded 1101869774 Bytes, about 16 MB/s
    stream load parquet:          31 seconds loaded 861443392 Bytes, about 26 MB/s
    insert into select:          29.1 seconds inserted 10000000 Rows, about 343K ops/s
    storage size: 17162263780 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [doris] github-actions[bot] commented on pull request #23172: [improvement](agg) Do not serialize bitmap to string

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #23172:
URL: https://github.com/apache/doris/pull/23172#issuecomment-1685028150

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [doris] github-actions[bot] commented on pull request #23172: [improvement](agg) Do not serialize bitmap to string

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #23172:
URL: https://github.com/apache/doris/pull/23172#issuecomment-1683938309

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [doris] hello-stephen commented on pull request #23172: [improvement](agg) Do not serialize bitmap to string

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #23172:
URL: https://github.com/apache/doris/pull/23172#issuecomment-1685039362

   (From new machine)TeamCity pipeline, clickbench performance test result:
    the sum of best hot time: 46.42 seconds
    stream load tsv:          542 seconds loaded 74807831229 Bytes, about 131 MB/s
    stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
    stream load orc:          65 seconds loaded 1101869774 Bytes, about 16 MB/s
    stream load parquet:          31 seconds loaded 861443392 Bytes, about 26 MB/s
    insert into select:          29.3 seconds inserted 10000000 Rows, about 341K ops/s
    storage size: 17162303525 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [doris] mrhhsg commented on pull request #23172: [improvement](agg) Do not serialize bitmap to string

Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #23172:
URL: https://github.com/apache/doris/pull/23172#issuecomment-1685026979

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [doris] HappenLee commented on a diff in pull request #23172: [improvement](agg) Do not serialize bitmap to string

Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on code in PR #23172:
URL: https://github.com/apache/doris/pull/23172#discussion_r1299145595


##########
be/src/vec/aggregate_functions/aggregate_function_bitmap.h:
##########
@@ -146,10 +146,127 @@ struct AggregateFunctionBitmapData {
     BitmapValue& get() { return value; }
 };
 
+template <typename Data, typename Derived>
+class AggregateFunctionBitmapSerializationHelper
+        : public IAggregateFunctionDataHelper<Data, Derived> {
+public:
+    using BaseHelper = IAggregateFunctionHelper<Derived>;
+
+    AggregateFunctionBitmapSerializationHelper(const DataTypes& argument_types_)
+            : IAggregateFunctionDataHelper<Data, Derived>(argument_types_) {}
+
+    void serialize_to_column(const std::vector<AggregateDataPtr>& places, size_t offset,
+                             MutableColumnPtr& dst, const size_t num_rows) const override {
+        if (version >= 3) {
+            auto& col = assert_cast<ColumnBitmap&>(*dst);
+            col.resize(num_rows);
+            auto* data = col.get_data().data();
+            for (size_t i = 0; i != num_rows; ++i) {
+                data[i] = this->data(places[i] + offset).value;
+            }
+        } else {
+            BaseHelper::serialize_to_column(places, offset, dst, num_rows);
+        }
+    }
+
+    void deserialize_and_merge_from_column(AggregateDataPtr __restrict place, const IColumn& column,
+                                           Arena* arena) const override {
+        if (version >= 3) {
+            auto& col = assert_cast<const ColumnBitmap&>(column);
+            const size_t num_rows = column.size();
+            auto* data = col.get_data().data();
+
+            for (size_t i = 0; i != num_rows; ++i) {
+                this->data(place).merge(data[i]);
+            }
+        } else {
+            BaseHelper::deserialize_and_merge_from_column(place, column, arena);
+        }
+    }
+
+    void deserialize_and_merge_from_column_range(AggregateDataPtr __restrict place,
+                                                 const IColumn& column, size_t begin, size_t end,
+                                                 Arena* arena) const override {
+        DCHECK(end <= column.size() && begin <= end)
+                << ", begin:" << begin << ", end:" << end << ", column.size():" << column.size();
+        if (version >= 3) {
+            auto& col = assert_cast<const ColumnBitmap&>(column);
+            auto* data = col.get_data().data();
+            for (size_t i = begin; i <= end; ++i) {
+                this->data(place).merge(data[i]);
+            }
+        } else {
+            BaseHelper::deserialize_and_merge_from_column_range(place, column, begin, end, arena);
+        }
+    }
+
+    void deserialize_and_merge_vec(const AggregateDataPtr* places, size_t offset,
+                                   AggregateDataPtr rhs, const ColumnString* column, Arena* arena,
+                                   const size_t num_rows) const override {
+        if (version >= 3) {
+            auto& col = assert_cast<const ColumnBitmap&>(*assert_cast<const IColumn*>(column));
+            auto* data = col.get_data().data();
+            for (size_t i = 0; i != num_rows; ++i) {
+                this->data(places[i]).merge(data[i]);
+            }
+        } else {
+            BaseHelper::deserialize_and_merge_vec(places, offset, rhs, column, arena, num_rows);
+        }
+    }
+
+    void deserialize_and_merge_vec_selected(const AggregateDataPtr* places, size_t offset,
+                                            AggregateDataPtr rhs, const ColumnString* column,
+                                            Arena* arena, const size_t num_rows) const override {
+        if (version >= 3) {
+            auto& col = assert_cast<const ColumnBitmap&>(*assert_cast<const IColumn*>(column));
+            auto* data = col.get_data().data();
+            for (size_t i = 0; i != num_rows; ++i) {
+                if (places[i]) {
+                    this->data(places[i]).merge(data[i]);
+                }
+            }
+        } else {
+            BaseHelper::deserialize_and_merge_vec_selected(places, offset, rhs, column, arena,
+                                                           num_rows);
+        }
+    }
+
+    void serialize_without_key_to_column(ConstAggregateDataPtr __restrict place,
+                                         IColumn& to) const override {
+        if (version >= 3) {
+            auto& col = assert_cast<ColumnBitmap&>(to);
+            size_t old_size = col.size();
+            col.resize(old_size + 1);
+            col.get_data()[old_size] = this->data(place).value;

Review Comment:
   seems better use std::move?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [doris] github-actions[bot] commented on pull request #23172: [improvement](agg) Do not serialize bitmap to string

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #23172:
URL: https://github.com/apache/doris/pull/23172#issuecomment-1683519286

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [doris] github-actions[bot] commented on pull request #23172: [improvement](agg) Do not serialize bitmap to string

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #23172:
URL: https://github.com/apache/doris/pull/23172#issuecomment-1685502900

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [doris] github-actions[bot] commented on pull request #23172: [improvement](agg) Do not serialize bitmap to string

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #23172:
URL: https://github.com/apache/doris/pull/23172#issuecomment-1684856813

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [doris] github-actions[bot] commented on pull request #23172: [improvement](agg) Do not serialize bitmap to string

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #23172:
URL: https://github.com/apache/doris/pull/23172#issuecomment-1683942368

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [doris] yiguolei merged pull request #23172: [improvement](agg) Do not serialize bitmap to string

Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei merged PR #23172:
URL: https://github.com/apache/doris/pull/23172


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [doris] mrhhsg commented on pull request #23172: [improvement](agg) Do not serialize bitmap to string

Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #23172:
URL: https://github.com/apache/doris/pull/23172#issuecomment-1683932216

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org