You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "taptao (via GitHub)" <gi...@apache.org> on 2024/03/06 00:38:53 UTC

[PR] [Improve](InPredict) enhance in predict with array type [doris]

taptao opened a new pull request, #31828:
URL: https://github.com/apache/doris/pull/31828

   ## Proposed changes
   
   Issue Number: close https://github.com/apache/doris/issues/31157
   
   Support in predict with array type.
   Example:select [1,2] in ([1,2],[3,4])
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Improve](InPredict) enhance in predict with array type [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31828:
URL: https://github.com/apache/doris/pull/31828#issuecomment-1987205368

   
   Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   ```
   Load test result on commit 75aee4ad0f3ec4a743d6ed86b68c6d93d517876a with default session variables
   Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
   Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
   Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
   Insert into select:       17.5 seconds inserted 10000000 Rows, about 571K ops/s
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Improve](InPredict) enhance in predict with array type [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31828:
URL: https://github.com/apache/doris/pull/31828#issuecomment-1987708681

   TeamCity be ut coverage result:
    Function Coverage: 35.00% (8556/24446) 
    Line Coverage: 26.83% (69533/259141)
    Region Coverage: 26.08% (36073/138337)
    Branch Coverage: 23.05% (18444/80032)
    Coverage Report: http://coverage.selectdb-in.cc/coverage/e009df429843d6f14dee1c9d644fe85a4a69b117_e009df429843d6f14dee1c9d644fe85a4a69b117/report/index.html


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Improve](InPredict) enhance in predict with array type [doris]

Posted by "amorynan (via GitHub)" <gi...@apache.org>.
amorynan commented on code in PR #31828:
URL: https://github.com/apache/doris/pull/31828#discussion_r1518706256


##########
be/src/vec/functions/function_array_in.h:
##########
@@ -0,0 +1,176 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+
+#pragma once
+
+#include <glog/logging.h>
+#include <stddef.h>
+
+#include <algorithm>
+#include <memory>
+#include <unordered_set>
+#include <utility>
+#include <vector>
+
+#include "common/status.h"
+#include "vec/columns/column.h"
+#include "vec/columns/column_const.h"
+#include "vec/columns/column_nullable.h"
+#include "vec/columns/column_struct.h"
+#include "vec/columns/column_vector.h"
+#include "vec/columns/columns_number.h"
+#include "vec/core/block.h"
+#include "vec/data_types/data_type_factory.hpp"
+#include "vec/data_types/data_type_nullable.h"
+#include "vec/data_types/data_type_number.h"
+#include "vec/functions/function.h"
+#include "vec/functions/function_struct_in.h"
+
+namespace doris {
+namespace vectorized {
+struct ColumnRowRef;
+}
+} // namespace doris
+
+namespace doris::vectorized {
+
+struct ArrayInState {

Review Comment:
   code is duplicated



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Improve](InPredict) enhance in predict with array type [doris]

Posted by "amorynan (via GitHub)" <gi...@apache.org>.
amorynan commented on code in PR #31828:
URL: https://github.com/apache/doris/pull/31828#discussion_r1519135511


##########
be/src/vec/functions/function_collection_in.h:
##########
@@ -115,36 +118,35 @@ class FunctionStructIn : public IFunction {
             const auto& [col, _] = unpack_if_const(const_column_ptr->column_ptr);
             if (col->is_nullable()) {
                 auto* null_col = vectorized::check_and_get_column<vectorized::ColumnNullable>(col);
-                if (null_col->has_null()) {
+                if (!state->null_in_set && null_col->has_null()) {

Review Comment:
   this if else logic maybe not right ! 
   think about this in params: (null, xxx, null), the second null will be inserted into args_column_ptr



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Improve](InPredict) enhance in predict with array type [doris]

Posted by "taptao (via GitHub)" <gi...@apache.org>.
taptao commented on PR #31828:
URL: https://github.com/apache/doris/pull/31828#issuecomment-1987670356

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Improve](InPredict) enhance in predict with array type [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31828:
URL: https://github.com/apache/doris/pull/31828#issuecomment-1987192026

   TeamCity be ut coverage result:
    Function Coverage: 35.36% (8555/24196) 
    Line Coverage: 27.15% (69533/256138)
    Region Coverage: 26.37% (36082/136836)
    Branch Coverage: 23.22% (18443/79444)
    Coverage Report: http://coverage.selectdb-in.cc/coverage/f773e040b642075a1d78708083ad3f2a8e0721cb_f773e040b642075a1d78708083ad3f2a8e0721cb/report/index.html


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Improve](InPredict) enhance in predict with array type [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31828:
URL: https://github.com/apache/doris/pull/31828#issuecomment-1987247064

   
   <details>
   <summary>ClickBench: <b>Total hot run time: 30.87 s</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
   ClickBench test result on commit 1624e1fd50f97670e5d5977a92c3b0eebd1dd69a, data reload: false
   
   query1	0.04	0.03	0.04
   query2	0.06	0.02	0.02
   query3	0.24	0.06	0.06
   query4	1.67	0.10	0.10
   query5	0.53	0.50	0.50
   query6	1.30	0.69	0.68
   query7	0.01	0.01	0.01
   query8	0.04	0.03	0.03
   query9	0.57	0.51	0.53
   query10	0.58	0.56	0.57
   query11	0.14	0.10	0.10
   query12	0.13	0.11	0.10
   query13	0.58	0.57	0.57
   query14	0.75	0.77	0.74
   query15	0.82	0.80	0.79
   query16	0.38	0.37	0.36
   query17	1.03	0.98	0.98
   query18	0.27	0.25	0.26
   query19	1.78	1.73	1.73
   query20	0.02	0.01	0.01
   query21	15.40	0.67	0.61
   query22	3.43	4.05	2.29
   query23	17.72	1.11	1.02
   query24	2.12	0.59	0.38
   query25	0.29	0.11	0.05
   query26	0.16	0.14	0.14
   query27	0.03	0.03	0.02
   query28	12.10	0.83	0.84
   query29	12.58	3.53	3.18
   query30	0.58	0.58	0.58
   query31	2.78	0.35	0.35
   query32	3.38	0.43	0.43
   query33	2.95	2.90	2.91
   query34	15.48	4.33	4.32
   query35	4.32	4.34	4.34
   query36	1.08	1.03	1.02
   query37	0.07	0.06	0.06
   query38	0.05	0.03	0.03
   query39	0.02	0.01	0.02
   query40	0.17	0.12	0.13
   query41	0.07	0.02	0.02
   query42	0.03	0.02	0.02
   query43	0.03	0.03	0.02
   Total cold run time: 105.78 s
   Total hot run time: 30.87 s
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Improve](InPredict) enhance in predict with array type [doris]

Posted by "amorynan (via GitHub)" <gi...@apache.org>.
amorynan commented on code in PR #31828:
URL: https://github.com/apache/doris/pull/31828#discussion_r1519136953


##########
be/src/vec/functions/function_collection_in.h:
##########
@@ -155,18 +157,25 @@ class FunctionStructIn : public IFunction {
 
         const ColumnWithTypeAndName& left_arg = block.get_by_position(arguments[0]);
         const auto& [materialized_column, col_const] = unpack_if_const(left_arg.column);
+        auto materialized_column_not_null = materialized_column;
+        if (materialized_column_not_null->is_nullable()) {
+            materialized_column_not_null = assert_cast<ColumnPtr>(
+                    vectorized::check_and_get_column<vectorized::ColumnNullable>(
+                            materialized_column_not_null)
+                            ->get_nested_column_ptr());
+        }
 
         for (size_t i = 0; i < input_rows_count; ++i) {
-            bool find = args_set.find({materialized_column, i}) != args_set.end();
+            bool find = args_set.find({materialized_column_not_null, i}) != args_set.end();
+
             if constexpr (negative) {
                 vec_res[i] = !find;
             } else {
                 vec_res[i] = find;
             }
-            if (in_state->null_in_set) {
-                vec_null_map_to[i] = negative == vec_res[i];
-            } else {
-                vec_null_map_to[i] = false;
+
+            if (!find && null_in_set) {

Review Comment:
   do  not make if not have else as possible as u can ! make code clear for read . and as less as condition in if , the compiler can make more optimize behavior. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Improve](InPredict) enhance in predict with array type [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31828:
URL: https://github.com/apache/doris/pull/31828#issuecomment-1988222016

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Improve](InPredict) enhance in predict with array type [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #31828:
URL: https://github.com/apache/doris/pull/31828#discussion_r1518583661


##########
be/src/vec/functions/function_array_in.h:
##########
@@ -0,0 +1,176 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+
+#pragma once
+
+#include <glog/logging.h>

Review Comment:
   warning: 'glog/logging.h' file not found [clang-diagnostic-error]
   ```cpp
   #include <glog/logging.h>
            ^
   ```
   



##########
be/src/vec/functions/function_array_in.h:
##########
@@ -0,0 +1,176 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+
+#pragma once
+
+#include <glog/logging.h>
+#include <stddef.h>
+
+#include <algorithm>
+#include <memory>
+#include <unordered_set>
+#include <utility>
+#include <vector>
+
+#include "common/status.h"
+#include "vec/columns/column.h"
+#include "vec/columns/column_const.h"
+#include "vec/columns/column_nullable.h"
+#include "vec/columns/column_struct.h"
+#include "vec/columns/column_vector.h"
+#include "vec/columns/columns_number.h"
+#include "vec/core/block.h"
+#include "vec/data_types/data_type_factory.hpp"
+#include "vec/data_types/data_type_nullable.h"
+#include "vec/data_types/data_type_number.h"
+#include "vec/functions/function.h"
+#include "vec/functions/function_struct_in.h"
+
+namespace doris {
+namespace vectorized {
+struct ColumnRowRef;
+}
+} // namespace doris
+
+namespace doris::vectorized {
+
+struct ArrayInState {
+    ENABLE_FACTORY_CREATOR(ArrayInState)
+    std::unordered_set<ColumnRowRef, ColumnRowRef> args_set;
+    bool null_in_set = false;
+};
+
+template <bool negative>
+class FunctionArrayIn : public IFunction {
+public:
+    static constexpr auto name = negative ? "array_not_in" : "array_in";
+
+    static FunctionPtr create() { return std::make_shared<FunctionArrayIn>(); }
+
+    String get_name() const override { return name; }
+
+    bool is_variadic() const override { return true; }
+
+    size_t get_number_of_arguments() const override { return 0; }
+
+    DataTypePtr get_return_type_impl(const DataTypes& args) const override {
+        for (const auto& arg : args) {
+            if (arg->is_nullable()) {
+                return make_nullable(std::make_shared<DataTypeUInt8>());
+            }
+        }
+        return std::make_shared<DataTypeUInt8>();
+    }
+
+    bool use_default_implementation_for_nulls() const override { return false; }
+
+    // make data in context into a set
+    Status open(FunctionContext* context, FunctionContext::FunctionStateScope scope) override {
+        if (scope == FunctionContext::THREAD_LOCAL) {
+            return Status::OK();
+        }
+
+        int num_args = context->get_num_args();
+        DCHECK(num_args >= 1);
+
+        const TypeDescriptor* arg_type_desc = context->get_arg_type(0);
+        DataTypePtr data_type_ptr =
+                DataTypeFactory::instance().create_data_type(*arg_type_desc, false);
+        MutableColumnPtr args_column_ptr = remove_nullable(data_type_ptr)->create_column();
+
+        std::shared_ptr<ArrayInState> array_in_state = std::make_shared<ArrayInState>();
+        context->set_function_state(FunctionContext::FRAGMENT_LOCAL, array_in_state);
+
+        // put const column to args_column
+        for (int i = 1; i < num_args; i++) {
+            ColumnPtrWrapper* const_col_ptr = context->get_constant_col(i);
+            if (const_col_ptr == nullptr) {
+                break;
+            }
+            const auto& [col, _] = unpack_if_const(const_col_ptr->column_ptr);
+            if (col->is_nullable()) {
+                auto* null_col = vectorized::check_and_get_column<vectorized::ColumnNullable>(col);
+                if (!array_in_state->null_in_set && null_col->has_null()) {
+                    array_in_state->null_in_set = true;
+                } else {
+                    args_column_ptr->insert_from(null_col->get_nested_column(), 0);
+                }
+            } else {
+                args_column_ptr->insert_from(*col, 0);
+            }
+        }
+        ColumnPtr column_ptr = std::move(args_column_ptr);
+        int col_size = column_ptr->size();
+        for (size_t i = 0; i < col_size; i++) {
+            array_in_state->args_set.insert({column_ptr, i});
+        }
+
+        return Status::OK();
+    }
+
+    Status execute_impl(FunctionContext* context, Block& block, const ColumnNumbers& arguments,
+                        size_t result, size_t input_rows_count) const override {
+        auto in_state = reinterpret_cast<ArrayInState*>(

Review Comment:
   warning: 'auto in_state' can be declared as 'auto *in_state' [readability-qualified-auto]
   
   ```suggestion
           auto *in_state = reinterpret_cast<ArrayInState*>(
   ```
   



##########
be/src/vec/functions/function_array_in.h:
##########
@@ -0,0 +1,176 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+
+#pragma once
+
+#include <glog/logging.h>
+#include <stddef.h>
+
+#include <algorithm>
+#include <memory>
+#include <unordered_set>
+#include <utility>
+#include <vector>
+
+#include "common/status.h"
+#include "vec/columns/column.h"
+#include "vec/columns/column_const.h"
+#include "vec/columns/column_nullable.h"
+#include "vec/columns/column_struct.h"
+#include "vec/columns/column_vector.h"
+#include "vec/columns/columns_number.h"
+#include "vec/core/block.h"
+#include "vec/data_types/data_type_factory.hpp"
+#include "vec/data_types/data_type_nullable.h"
+#include "vec/data_types/data_type_number.h"
+#include "vec/functions/function.h"
+#include "vec/functions/function_struct_in.h"
+
+namespace doris {
+namespace vectorized {

Review Comment:
   warning: nested namespaces can be concatenated [modernize-concat-nested-namespaces]
   
   ```suggestion
   namespace doris::vectorized {
   ```
   
   be/src/vec/functions/function_array_in.h:46:
   ```diff
   - }
   - } // namespace doris
   + } // namespace doris
   ```
   



##########
be/src/vec/functions/function_array_in.h:
##########
@@ -0,0 +1,176 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+
+#pragma once
+
+#include <glog/logging.h>
+#include <stddef.h>

Review Comment:
   warning: inclusion of deprecated C++ header 'stddef.h'; consider using 'cstddef' instead [modernize-deprecated-headers]
   
   ```suggestion
   #include <cstddef>
   ```
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Improve](InPredict) enhance in predict with array type [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31828:
URL: https://github.com/apache/doris/pull/31828#issuecomment-1987198909

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 38085 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit 75aee4ad0f3ec4a743d6ed86b68c6d93d517876a, data reload: false
   
   ------ Round 1 ----------------------------------
   q1	17585	4126	4011	4011
   q2	2030	148	142	142
   q3	10668	945	941	941
   q4	4703	956	956	956
   q5	7589	2871	3021	2871
   q6	180	121	129	121
   q7	1282	831	818	818
   q8	9593	2072	2078	2072
   q9	7271	6505	6457	6457
   q10	8235	2586	2562	2562
   q11	423	218	221	218
   q12	739	327	324	324
   q13	17971	2933	2949	2933
   q14	291	262	254	254
   q15	498	441	441	441
   q16	496	404	399	399
   q17	957	907	890	890
   q18	6815	5971	5894	5894
   q19	1569	1523	1519	1519
   q20	542	297	300	297
   q21	7546	3691	3673	3673
   q22	853	292	297	292
   Total cold run time: 107836 ms
   Total hot run time: 38085 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   q1	4008	3964	4005	3964
   q2	333	235	227	227
   q3	2990	2918	2969	2918
   q4	1847	1842	1810	1810
   q5	5241	5292	5246	5246
   q6	212	115	115	115
   q7	2264	1838	1861	1838
   q8	3223	3306	3291	3291
   q9	8558	8584	8546	8546
   q10	6165	3734	3690	3690
   q11	527	460	446	446
   q12	704	559	546	546
   q13	6804	2863	2829	2829
   q14	280	254	259	254
   q15	485	433	438	433
   q16	456	428	421	421
   q17	1689	1668	1665	1665
   q18	7915	7426	7310	7310
   q19	1630	1609	1615	1609
   q20	1912	1732	1748	1732
   q21	5067	4821	4771	4771
   q22	570	478	474	474
   Total cold run time: 62880 ms
   Total hot run time: 54135 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Improve](InPredict) enhance in predict with array type [doris]

Posted by "taptao (via GitHub)" <gi...@apache.org>.
taptao commented on PR #31828:
URL: https://github.com/apache/doris/pull/31828#issuecomment-1987180370

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Improve](InPredict) enhance in predict with array type [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31828:
URL: https://github.com/apache/doris/pull/31828#issuecomment-1987710199

   
   <details>
   <summary>TPC-DS: <b>Total hot run time: 186213 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit e009df429843d6f14dee1c9d644fe85a4a69b117, data reload: false
   
   query1	927	356	341	341
   query2	7381	1998	1953	1953
   query3	6704	217	219	217
   query4	24918	20994	20946	20946
   query5	4315	1441	502	502
   query6	269	178	202	178
   query7	4627	320	312	312
   query8	243	189	171	171
   query9	8437	2185	2182	2182
   query10	443	240	248	240
   query11	15025	14553	14444	14444
   query12	136	89	87	87
   query13	1624	442	418	418
   query14	10195	10016	9620	9620
   query15	245	201	196	196
   query16	7784	276	278	276
   query17	1697	631	572	572
   query18	1976	292	293	292
   query19	206	169	171	169
   query20	117	87	96	87
   query21	200	130	125	125
   query22	4576	4427	4419	4419
   query23	31535	31014	30924	30924
   query24	11527	3039	3042	3039
   query25	695	399	384	384
   query26	1542	169	169	169
   query27	3084	368	356	356
   query28	6708	1812	1806	1806
   query29	1047	610	598	598
   query30	301	152	154	152
   query31	927	739	728	728
   query32	112	65	62	62
   query33	742	272	265	265
   query34	970	474	471	471
   query35	862	647	641	641
   query36	913	809	814	809
   query37	158	79	77	77
   query38	3303	3161	3133	3133
   query39	1436	1368	1388	1368
   query40	230	126	143	126
   query41	68	62	62	62
   query42	114	102	106	102
   query43	437	398	416	398
   query44	1099	721	716	716
   query45	273	252	236	236
   query46	1049	801	786	786
   query47	1626	1572	1561	1561
   query48	431	345	340	340
   query49	1155	341	365	341
   query50	779	380	383	380
   query51	6689	6506	6683	6506
   query52	108	98	99	98
   query53	365	296	285	285
   query54	349	264	254	254
   query55	82	86	84	84
   query56	257	227	230	227
   query57	1084	999	997	997
   query58	256	223	238	223
   query59	2614	2441	2404	2404
   query60	277	292	280	280
   query61	116	112	112	112
   query62	665	419	417	417
   query63	313	286	295	286
   query64	6612	3453	3130	3130
   query65	3077	3030	3010	3010
   query66	1383	325	316	316
   query67	14957	14274	14777	14274
   query68	9301	573	581	573
   query69	661	404	397	397
   query70	1354	1104	1130	1104
   query71	520	301	284	284
   query72	6526	2656	2531	2531
   query73	1567	337	316	316
   query74	6884	6563	6492	6492
   query75	9595	8035	8018	8018
   query76	5604	1068	1160	1068
   query77	645	260	250	250
   query78	10271	9534	9585	9534
   query79	10846	522	528	522
   query80	1390	402	445	402
   query81	481	210	215	210
   query82	398	195	208	195
   query83	220	151	149	149
   query84	284	78	83	78
   query85	1114	342	332	332
   query86	390	300	303	300
   query87	3436	3222	3250	3222
   query88	4216	2294	2282	2282
   query89	539	369	368	368
   query90	2024	180	187	180
   query91	164	130	125	125
   query92	59	47	49	47
   query93	6264	534	523	523
   query94	1329	189	188	188
   query95	452	349	352	349
   query96	613	270	261	261
   query97	4406	4257	4282	4257
   query98	234	216	212	212
   query99	1151	770	764	764
   Total cold run time: 307310 ms
   Total hot run time: 186213 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Improve](InPredict) enhance in predict with array type [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31828:
URL: https://github.com/apache/doris/pull/31828#issuecomment-1987202398

   
   <details>
   <summary>TPC-DS: <b>Total hot run time: 187069 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit 75aee4ad0f3ec4a743d6ed86b68c6d93d517876a, data reload: false
   
   query1	936	358	339	339
   query2	7389	2104	2023	2023
   query3	6706	212	211	211
   query4	26495	21030	21062	21030
   query5	4292	457	481	457
   query6	270	180	178	178
   query7	4621	321	317	317
   query8	233	180	178	178
   query9	8323	2291	2299	2291
   query10	473	239	244	239
   query11	14965	14563	14201	14201
   query12	135	92	89	89
   query13	1659	434	444	434
   query14	10597	9698	9829	9698
   query15	246	204	201	201
   query16	7956	286	278	278
   query17	1513	589	571	571
   query18	1971	291	295	291
   query19	201	174	166	166
   query20	101	90	92	90
   query21	201	140	133	133
   query22	4658	4457	4413	4413
   query23	31583	30957	30917	30917
   query24	11662	3122	3084	3084
   query25	678	413	394	394
   query26	1661	173	165	165
   query27	3049	372	372	372
   query28	6389	1889	1866	1866
   query29	1089	607	603	603
   query30	308	150	147	147
   query31	922	748	743	743
   query32	106	69	68	68
   query33	776	261	257	257
   query34	1038	482	489	482
   query35	851	637	643	637
   query36	926	815	842	815
   query37	142	74	80	74
   query38	3220	3117	3133	3117
   query39	1412	1386	1383	1383
   query40	301	118	114	114
   query41	63	55	56	55
   query42	111	105	103	103
   query43	439	396	403	396
   query44	1099	721	729	721
   query45	283	257	254	254
   query46	1054	807	778	778
   query47	1648	1553	1551	1551
   query48	427	357	357	357
   query49	1236	364	343	343
   query50	784	381	383	381
   query51	6852	6702	6652	6652
   query52	103	99	97	97
   query53	350	286	292	286
   query54	308	273	263	263
   query55	88	86	81	81
   query56	260	222	234	222
   query57	1079	1002	979	979
   query58	248	227	229	227
   query59	2502	2281	2413	2281
   query60	267	254	261	254
   query61	135	117	117	117
   query62	643	422	413	413
   query63	309	285	280	280
   query64	5951	3209	3144	3144
   query65	3051	3009	3007	3007
   query66	1088	334	317	317
   query67	15131	14690	14517	14517
   query68	16309	548	584	548
   query69	748	389	379	379
   query70	1937	1065	1087	1065
   query71	593	272	267	267
   query72	10273	2619	2479	2479
   query73	3974	338	345	338
   query74	7248	6905	6979	6905
   query75	12378	8013	8066	8013
   query76	7351	1113	1193	1113
   query77	1177	258	250	250
   query78	10206	9758	9608	9608
   query79	8963	503	510	503
   query80	938	384	385	384
   query81	489	215	207	207
   query82	332	195	196	195
   query83	413	139	135	135
   query84	286	84	78	78
   query85	1226	337	331	331
   query86	372	307	296	296
   query87	3437	3216	3197	3197
   query88	3175	2376	2351	2351
   query89	527	352	362	352
   query90	2288	171	170	170
   query91	165	130	134	130
   query92	63	50	51	50
   query93	2840	537	533	533
   query94	1499	192	189	189
   query95	459	345	338	338
   query96	588	267	263	263
   query97	4429	4279	4311	4279
   query98	223	220	221	220
   query99	1054	790	761	761
   Total cold run time: 321523 ms
   Total hot run time: 187069 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Improve](InPredict) enhance in predict with array type [doris]

Posted by "taptao (via GitHub)" <gi...@apache.org>.
taptao commented on PR #31828:
URL: https://github.com/apache/doris/pull/31828#issuecomment-1987194166

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Improve](InPredict) enhance in predict with array type [doris]

Posted by "taptao (via GitHub)" <gi...@apache.org>.
taptao commented on code in PR #31828:
URL: https://github.com/apache/doris/pull/31828#discussion_r1519034861


##########
be/src/vec/functions/function_array_in.h:
##########
@@ -0,0 +1,176 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+
+#pragma once
+
+#include <glog/logging.h>
+#include <stddef.h>
+
+#include <algorithm>
+#include <memory>
+#include <unordered_set>
+#include <utility>
+#include <vector>
+
+#include "common/status.h"
+#include "vec/columns/column.h"
+#include "vec/columns/column_const.h"
+#include "vec/columns/column_nullable.h"
+#include "vec/columns/column_struct.h"
+#include "vec/columns/column_vector.h"
+#include "vec/columns/columns_number.h"
+#include "vec/core/block.h"
+#include "vec/data_types/data_type_factory.hpp"
+#include "vec/data_types/data_type_nullable.h"
+#include "vec/data_types/data_type_number.h"
+#include "vec/functions/function.h"
+#include "vec/functions/function_struct_in.h"
+
+namespace doris {
+namespace vectorized {
+struct ColumnRowRef;
+}
+} // namespace doris
+
+namespace doris::vectorized {
+
+struct ArrayInState {

Review Comment:
   done!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Improve](InPredict) enhance in predict with array type [doris]

Posted by "taptao (via GitHub)" <gi...@apache.org>.
taptao commented on PR #31828:
URL: https://github.com/apache/doris/pull/31828#issuecomment-1986879502

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Improve](InPredict) enhance in predict with array type [doris]

Posted by "amorynan (via GitHub)" <gi...@apache.org>.
amorynan commented on PR #31828:
URL: https://github.com/apache/doris/pull/31828#issuecomment-1987044630

   just keep a function_collection_in.h/cpp file to handle array/struct/map this collection type. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Improve](InPredict) enhance in predict with array type [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31828:
URL: https://github.com/apache/doris/pull/31828#issuecomment-1987239281

   TeamCity be ut coverage result:
    Function Coverage: 35.36% (8555/24196) 
    Line Coverage: 27.14% (69523/256138)
    Region Coverage: 26.37% (36078/136836)
    Branch Coverage: 23.22% (18443/79444)
    Coverage Report: http://coverage.selectdb-in.cc/coverage/1624e1fd50f97670e5d5977a92c3b0eebd1dd69a_1624e1fd50f97670e5d5977a92c3b0eebd1dd69a/report/index.html


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Improve](InPredict) enhance in predict with array type [doris]

Posted by "taptao (via GitHub)" <gi...@apache.org>.
taptao commented on PR #31828:
URL: https://github.com/apache/doris/pull/31828#issuecomment-1987227370

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Improve](InPredict) enhance in predict with array type [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31828:
URL: https://github.com/apache/doris/pull/31828#issuecomment-1987202787

   TeamCity be ut coverage result:
    Function Coverage: 35.36% (8555/24196) 
    Line Coverage: 27.14% (69517/256138)
    Region Coverage: 26.36% (36066/136836)
    Branch Coverage: 23.21% (18439/79444)
    Coverage Report: http://coverage.selectdb-in.cc/coverage/75aee4ad0f3ec4a743d6ed86b68c6d93d517876a_75aee4ad0f3ec4a743d6ed86b68c6d93d517876a/report/index.html


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Improve](InPredict) enhance in predict with array type [doris]

Posted by "amorynan (via GitHub)" <gi...@apache.org>.
amorynan commented on code in PR #31828:
URL: https://github.com/apache/doris/pull/31828#discussion_r1519131053


##########
be/src/vec/functions/function_collection_in.h:
##########
@@ -98,14 +98,17 @@ class FunctionStructIn : public IFunction {
         if (scope == FunctionContext::THREAD_LOCAL) {
             return Status::OK();
         }
-        std::shared_ptr<StructInState> state = std::make_shared<StructInState>();
+        int num_args = context->get_num_args();
+        DCHECK(num_args >= 1);
+
+        std::shared_ptr<CollectionInState> state = std::make_shared<CollectionInState>();
         context->set_function_state(scope, state);
-        DCHECK(context->get_num_args() >= 1);
+
         auto* col_desc = context->get_arg_type(0);
-        DataTypePtr args_type = DataTypeFactory::instance().create_data_type(*col_desc);
-        MutableColumnPtr column_struct_ptr_args = remove_nullable(args_type)->create_column();
-        NullMap null_map(context->get_num_args(), false);
-        for (int i = 1; i < context->get_num_args(); ++i) {
+        DataTypePtr args_type = DataTypeFactory::instance().create_data_type(*col_desc, false);
+        MutableColumnPtr args_column_ptr = remove_nullable(args_type)->create_column();

Review Comment:
   create_data_type(*col_desc, false); u set false already , there is no need to remove_nullable again



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Improve](InPredict) enhance in predict with array type [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31828:
URL: https://github.com/apache/doris/pull/31828#issuecomment-1988093525

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Improve](InPredict) enhance in predict with array type [doris]

Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee merged PR #31828:
URL: https://github.com/apache/doris/pull/31828


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Improve](InPredict) enhance in predict with array type [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31828:
URL: https://github.com/apache/doris/pull/31828#issuecomment-1986892972

   TeamCity be ut coverage result:
    Function Coverage: 35.35% (8556/24205) 
    Line Coverage: 27.13% (69522/256232)
    Region Coverage: 26.35% (36073/136885)
    Branch Coverage: 23.20% (18435/79470)
    Coverage Report: http://coverage.selectdb-in.cc/coverage/d9f8c5446d7dc66b23313b44a7ea4891483bdb95_d9f8c5446d7dc66b23313b44a7ea4891483bdb95/report/index.html


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Improve](InPredict) enhance in predict with array type [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31828:
URL: https://github.com/apache/doris/pull/31828#issuecomment-1987698954

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 38520 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit e009df429843d6f14dee1c9d644fe85a4a69b117, data reload: false
   
   ------ Round 1 ----------------------------------
   q1	18541	4211	4180	4180
   q2	2751	159	150	150
   q3	11305	975	971	971
   q4	5242	971	974	971
   q5	8161	2954	3084	2954
   q6	181	125	129	125
   q7	1349	867	873	867
   q8	9993	2073	2065	2065
   q9	7300	6499	6483	6483
   q10	8216	2583	2561	2561
   q11	425	229	224	224
   q12	770	325	317	317
   q13	17951	3014	3014	3014
   q14	277	253	249	249
   q15	508	482	459	459
   q16	487	404	408	404
   q17	942	870	851	851
   q18	6625	5900	5914	5900
   q19	1570	1531	1512	1512
   q20	551	286	286	286
   q21	7536	3677	3724	3677
   q22	785	300	301	300
   Total cold run time: 111466 ms
   Total hot run time: 38520 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   q1	4010	4032	4018	4018
   q2	321	231	228	228
   q3	2959	2886	2954	2886
   q4	1848	1838	1881	1838
   q5	5276	5292	5277	5277
   q6	205	116	120	116
   q7	2283	1865	1828	1828
   q8	3216	3303	3295	3295
   q9	8538	8558	8563	8558
   q10	6141	3721	3720	3720
   q11	548	440	440	440
   q12	701	563	555	555
   q13	3643	2891	2839	2839
   q14	273	257	260	257
   q15	498	462	448	448
   q16	457	430	405	405
   q17	1687	1674	1664	1664
   q18	7961	7439	7324	7324
   q19	1655	1624	1625	1624
   q20	1913	1693	1713	1693
   q21	4998	4774	4803	4774
   q22	540	447	442	442
   Total cold run time: 59671 ms
   Total hot run time: 54229 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Improve](InPredict) enhance in predict with array type [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31828:
URL: https://github.com/apache/doris/pull/31828#issuecomment-1987247911

   
   Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   ```
   Load test result on commit 1624e1fd50f97670e5d5977a92c3b0eebd1dd69a with default session variables
   Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
   Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
   Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
   Insert into select:       16.0 seconds inserted 10000000 Rows, about 625K ops/s
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Improve](InPredict) enhance in predict with array type [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31828:
URL: https://github.com/apache/doris/pull/31828#issuecomment-1987204427

   
   <details>
   <summary>ClickBench: <b>Total hot run time: 30.59 s</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
   ClickBench test result on commit 75aee4ad0f3ec4a743d6ed86b68c6d93d517876a, data reload: false
   
   query1	0.04	0.03	0.03
   query2	0.06	0.02	0.02
   query3	0.23	0.06	0.06
   query4	1.67	0.09	0.10
   query5	0.50	0.50	0.51
   query6	1.24	0.68	0.68
   query7	0.02	0.01	0.02
   query8	0.04	0.03	0.03
   query9	0.56	0.52	0.51
   query10	0.58	0.58	0.57
   query11	0.13	0.10	0.10
   query12	0.12	0.10	0.10
   query13	0.58	0.57	0.57
   query14	0.73	0.75	0.75
   query15	0.82	0.80	0.81
   query16	0.38	0.40	0.39
   query17	0.97	1.00	0.97
   query18	0.25	0.25	0.26
   query19	1.82	1.74	1.68
   query20	0.01	0.02	0.01
   query21	15.43	0.63	0.60
   query22	3.24	3.65	2.28
   query23	17.30	1.02	0.96
   query24	2.12	0.38	0.19
   query25	0.27	0.12	0.05
   query26	0.17	0.14	0.14
   query27	0.03	0.04	0.03
   query28	12.53	0.84	0.84
   query29	12.56	3.23	3.29
   query30	0.59	0.58	0.58
   query31	2.78	0.35	0.34
   query32	3.39	0.44	0.44
   query33	2.98	2.90	2.90
   query34	15.49	4.31	4.29
   query35	4.31	4.32	4.28
   query36	1.09	1.01	1.04
   query37	0.07	0.05	0.06
   query38	0.04	0.04	0.03
   query39	0.02	0.02	0.02
   query40	0.18	0.14	0.14
   query41	0.07	0.02	0.02
   query42	0.02	0.02	0.02
   query43	0.02	0.03	0.03
   Total cold run time: 105.45 s
   Total hot run time: 30.59 s
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Improve](InPredict) enhance in predict with array type [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31828:
URL: https://github.com/apache/doris/pull/31828#issuecomment-1987245037

   
   <details>
   <summary>TPC-DS: <b>Total hot run time: 187160 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit 1624e1fd50f97670e5d5977a92c3b0eebd1dd69a, data reload: false
   
   query1	910	342	337	337
   query2	7374	2153	2072	2072
   query3	6704	217	207	207
   query4	26687	21146	21152	21146
   query5	4315	471	472	471
   query6	279	183	179	179
   query7	4627	311	302	302
   query8	233	174	168	168
   query9	8394	2296	2291	2291
   query10	458	245	250	245
   query11	14956	14639	14288	14288
   query12	135	90	87	87
   query13	1660	451	437	437
   query14	11164	10175	9849	9849
   query15	267	200	200	200
   query16	8111	287	284	284
   query17	1709	581	552	552
   query18	2102	280	266	266
   query19	204	160	161	160
   query20	101	92	89	89
   query21	203	129	126	126
   query22	4596	4432	4417	4417
   query23	30953	30995	30514	30514
   query24	10659	3083	3011	3011
   query25	643	371	374	371
   query26	1501	165	166	165
   query27	3035	378	362	362
   query28	6505	1863	1861	1861
   query29	1001	621	613	613
   query30	310	161	150	150
   query31	928	736	762	736
   query32	107	64	62	62
   query33	766	274	277	274
   query34	984	474	498	474
   query35	851	641	630	630
   query36	933	841	847	841
   query37	151	86	89	86
   query38	3164	3159	3128	3128
   query39	1431	1388	1387	1387
   query40	220	132	125	125
   query41	61	55	54	54
   query42	113	105	104	104
   query43	430	398	402	398
   query44	1107	732	716	716
   query45	296	250	258	250
   query46	1071	836	784	784
   query47	1627	1568	1537	1537
   query48	439	351	358	351
   query49	1167	357	352	352
   query50	794	396	400	396
   query51	6736	6706	6707	6706
   query52	103	95	94	94
   query53	363	293	286	286
   query54	297	259	279	259
   query55	86	85	85	85
   query56	248	238	229	229
   query57	1072	1002	993	993
   query58	248	231	236	231
   query59	2500	2405	2394	2394
   query60	311	288	257	257
   query61	135	117	116	116
   query62	663	407	407	407
   query63	309	283	283	283
   query64	5055	3465	3568	3465
   query65	3041	3026	3034	3026
   query66	944	370	351	351
   query67	15048	14505	14267	14267
   query68	12377	578	589	578
   query69	761	406	394	394
   query70	1376	1158	1120	1120
   query71	630	294	297	294
   query72	10224	2634	2451	2451
   query73	1996	341	346	341
   query74	7247	6841	6739	6739
   query75	11878	7916	7969	7916
   query76	8104	1100	1188	1100
   query77	989	273	271	271
   query78	10214	9568	9489	9489
   query79	10471	542	545	542
   query80	1170	404	462	404
   query81	486	216	205	205
   query82	333	202	200	200
   query83	240	148	143	143
   query84	279	81	78	78
   query85	1234	340	329	329
   query86	365	316	276	276
   query87	3418	3184	3186	3184
   query88	3474	2375	2380	2375
   query89	539	365	363	363
   query90	2354	180	180	180
   query91	164	127	128	127
   query92	59	53	51	51
   query93	4742	545	545	545
   query94	1610	205	193	193
   query95	464	359	355	355
   query96	607	268	267	267
   query97	4421	4327	4312	4312
   query98	224	233	215	215
   query99	1083	777	769	769
   Total cold run time: 317128 ms
   Total hot run time: 187160 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Improve](InPredict) enhance in predict with array type [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31828:
URL: https://github.com/apache/doris/pull/31828#issuecomment-1987241156

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 37535 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit 1624e1fd50f97670e5d5977a92c3b0eebd1dd69a, data reload: false
   
   ------ Round 1 ----------------------------------
   q1	17641	3995	3977	3977
   q2	2016	142	138	138
   q3	10568	966	923	923
   q4	4656	919	934	919
   q5	7587	2850	2973	2850
   q6	176	126	125	125
   q7	1267	822	810	810
   q8	9482	1976	2045	1976
   q9	7248	6417	6385	6385
   q10	8211	2558	2550	2550
   q11	415	223	215	215
   q12	796	326	317	317
   q13	17985	2918	2908	2908
   q14	275	248	251	248
   q15	487	447	440	440
   q16	489	388	392	388
   q17	938	863	825	825
   q18	6609	5878	5794	5794
   q19	1548	1511	1510	1510
   q20	533	289	280	280
   q21	7547	3674	3670	3670
   q22	788	292	287	287
   Total cold run time: 107262 ms
   Total hot run time: 37535 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   q1	3991	3950	3975	3950
   q2	320	229	222	222
   q3	2947	2905	2881	2881
   q4	1867	1772	1765	1765
   q5	5234	5209	5194	5194
   q6	200	116	118	116
   q7	2256	1814	1820	1814
   q8	3174	3297	3248	3248
   q9	8505	8466	8522	8466
   q10	6134	3725	3692	3692
   q11	530	450	433	433
   q12	696	564	531	531
   q13	15050	2765	2784	2765
   q14	291	264	252	252
   q15	476	441	437	437
   q16	453	425	423	423
   q17	1674	1649	1655	1649
   q18	7875	7432	7314	7314
   q19	1801	1598	1607	1598
   q20	1928	1786	1690	1690
   q21	4965	4745	4840	4745
   q22	544	471	470	470
   Total cold run time: 70911 ms
   Total hot run time: 53655 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Improve](InPredict) enhance in predict with array type [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31828:
URL: https://github.com/apache/doris/pull/31828#issuecomment-1987715550

   
   <details>
   <summary>ClickBench: <b>Total hot run time: 29.78 s</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
   ClickBench test result on commit e009df429843d6f14dee1c9d644fe85a4a69b117, data reload: false
   
   query1	0.04	0.03	0.04
   query2	0.06	0.03	0.02
   query3	0.24	0.06	0.06
   query4	1.66	0.09	0.10
   query5	0.50	0.49	0.50
   query6	1.21	0.68	0.67
   query7	0.02	0.01	0.01
   query8	0.03	0.03	0.03
   query9	0.58	0.51	0.54
   query10	0.58	0.58	0.56
   query11	0.14	0.09	0.10
   query12	0.13	0.11	0.10
   query13	0.58	0.58	0.57
   query14	0.76	0.74	0.77
   query15	0.83	0.81	0.82
   query16	0.37	0.36	0.39
   query17	0.98	1.03	1.01
   query18	0.25	0.25	0.25
   query19	1.86	1.75	1.69
   query20	0.01	0.01	0.01
   query21	15.41	0.61	0.59
   query22	3.48	3.56	1.64
   query23	17.28	1.00	0.86
   query24	2.19	0.40	0.30
   query25	0.22	0.11	0.07
   query26	0.18	0.13	0.14
   query27	0.03	0.03	0.03
   query28	12.36	0.87	0.83
   query29	12.54	3.20	3.28
   query30	0.60	0.60	0.55
   query31	2.80	0.34	0.34
   query32	3.39	0.44	0.44
   query33	2.78	2.81	2.82
   query34	15.48	4.30	4.29
   query35	4.33	4.30	4.31
   query36	1.08	1.01	1.02
   query37	0.07	0.06	0.05
   query38	0.04	0.04	0.03
   query39	0.02	0.02	0.02
   query40	0.19	0.13	0.14
   query41	0.08	0.02	0.02
   query42	0.03	0.03	0.02
   query43	0.02	0.02	0.02
   Total cold run time: 105.43 s
   Total hot run time: 29.78 s
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Improve](InPredict) enhance in predict with array type [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31828:
URL: https://github.com/apache/doris/pull/31828#issuecomment-1987192629

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 37658 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit f773e040b642075a1d78708083ad3f2a8e0721cb, data reload: false
   
   ------ Round 1 ----------------------------------
   q1	17633	4118	4013	4013
   q2	2037	145	138	138
   q3	10571	968	924	924
   q4	4658	937	960	937
   q5	7607	2841	2979	2841
   q6	177	122	123	122
   q7	1306	825	836	825
   q8	9478	1946	2081	1946
   q9	7255	6480	6419	6419
   q10	8214	2552	2563	2552
   q11	417	221	211	211
   q12	750	319	308	308
   q13	17970	2941	2916	2916
   q14	278	261	259	259
   q15	487	459	444	444
   q16	485	391	404	391
   q17	955	865	847	847
   q18	6625	6020	5827	5827
   q19	1554	1521	1515	1515
   q20	571	278	274	274
   q21	7573	3676	3661	3661
   q22	789	288	330	288
   Total cold run time: 107390 ms
   Total hot run time: 37658 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   q1	4012	3970	3964	3964
   q2	311	218	229	218
   q3	2939	2922	2907	2907
   q4	1824	1781	1825	1781
   q5	5205	5264	5236	5236
   q6	202	116	114	114
   q7	2249	1827	1814	1814
   q8	3226	3261	3275	3261
   q9	8531	8522	8479	8479
   q10	6152	3745	3719	3719
   q11	525	442	441	441
   q12	696	533	545	533
   q13	14842	2802	2830	2802
   q14	281	255	256	255
   q15	472	442	442	442
   q16	454	419	411	411
   q17	1692	1672	1641	1641
   q18	7777	7377	7339	7339
   q19	1632	1596	1602	1596
   q20	1986	1717	1769	1717
   q21	4990	4752	4697	4697
   q22	531	470	476	470
   Total cold run time: 70529 ms
   Total hot run time: 53837 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org