You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "zclllyybb (via GitHub)" <gi...@apache.org> on 2023/11/30 22:46:39 UTC

[PR] [Feature](functions) support ignore and nullable functions [doris]

zclllyybb opened a new pull request, #27848:
URL: https://github.com/apache/doris/pull/27848

   ## Proposed changes
   
   Issue Number: close #xxx
   
   <!--Describe your changes.-->
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Feature](functions) support ignore and nullable functions [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27848:
URL: https://github.com/apache/doris/pull/27848#issuecomment-1836416302

   (From new machine)TeamCity pipeline, clickbench performance test result:
    the sum of best hot time: 43.76 seconds
    stream load tsv:          561 seconds loaded 74807831229 Bytes, about 127 MB/s
    stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
    stream load orc:          65 seconds loaded 1101869774 Bytes, about 16 MB/s
    stream load parquet:          32 seconds loaded 861443392 Bytes, about 25 MB/s
    insert into select:          28.5 seconds inserted 10000000 Rows, about 350K ops/s
    storage size: 17163749303 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Feature](functions) support ignore and nullable functions [doris]

Posted by "zclllyybb (via GitHub)" <gi...@apache.org>.
zclllyybb commented on PR #27848:
URL: https://github.com/apache/doris/pull/27848#issuecomment-1836218860

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Feature](functions) support ignore and nullable functions [doris]

Posted by "zclllyybb (via GitHub)" <gi...@apache.org>.
zclllyybb commented on PR #27848:
URL: https://github.com/apache/doris/pull/27848#issuecomment-1839935771

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Feature](functions) support ignore and nullable functions [doris]

Posted by "zclllyybb (via GitHub)" <gi...@apache.org>.
zclllyybb commented on PR #27848:
URL: https://github.com/apache/doris/pull/27848#issuecomment-1834679565

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Feature](functions) support ignore and nullable functions [doris]

Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow commented on code in PR #27848:
URL: https://github.com/apache/doris/pull/27848#discussion_r1411506223


##########
docs/zh-CN/docs/sql-manual/sql-functions/debug-functions/non-nullable.md:
##########
@@ -0,0 +1,56 @@
+---
+{
+    "title": "NON_NULLABLE",
+    "language": "zh-CN"
+}
+---
+
+<!-- 
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+## non_nullable
+
+### description
+#### Syntax
+
+`non_nullable(expr)`
+
+如果`expr`为nullable类型,则返回值为该表达式的非nullable值;否则报错。

Review Comment:
   为啥要报错呢?不变就可以了吧。另外,如果该列中有null如何处理?我看下面把null变成0,这是不可接受的行为,会导致数据错误



##########
docs/zh-CN/docs/sql-manual/sql-functions/debug-functions/non-nullable.md:
##########
@@ -0,0 +1,56 @@
+---
+{
+    "title": "NON_NULLABLE",
+    "language": "zh-CN"
+}
+---
+
+<!-- 
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+## non_nullable
+
+### description
+#### Syntax
+
+`non_nullable(expr)`
+
+如果`expr`为nullable类型,则返回值为该表达式的非nullable值;否则报错。

Review Comment:
   为啥要报错呢?不变就可以了吧。另外,如果该列中有null如何处理?我看下面把null变成0,这是不可接受的行为,会导致数据错误



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Feature](functions) support ignore and nullable functions [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27848:
URL: https://github.com/apache/doris/pull/27848#issuecomment-1838040230

   (From new machine)TeamCity pipeline, clickbench performance test result:
    the sum of best hot time: 43.51 seconds
    stream load tsv:          561 seconds loaded 74807831229 Bytes, about 127 MB/s
    stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
    stream load orc:          65 seconds loaded 1101869774 Bytes, about 16 MB/s
    stream load parquet:          32 seconds loaded 861443392 Bytes, about 25 MB/s
    insert into select:          28.8 seconds inserted 10000000 Rows, about 347K ops/s
    storage size: 17163785888 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Feature](functions) support ignore and nullable functions [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27848:
URL: https://github.com/apache/doris/pull/27848#issuecomment-1835380884

   (From new machine)TeamCity pipeline, clickbench performance test result:
    the sum of best hot time: 44.15 seconds
    stream load tsv:          562 seconds loaded 74807831229 Bytes, about 126 MB/s
    stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
    stream load orc:          65 seconds loaded 1101869774 Bytes, about 16 MB/s
    stream load parquet:          32 seconds loaded 861443392 Bytes, about 25 MB/s
    insert into select:          28.8 seconds inserted 10000000 Rows, about 347K ops/s
    storage size: 17163755375 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Feature](functions) support ignore and nullable functions [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #27848:
URL: https://github.com/apache/doris/pull/27848#discussion_r1411383112


##########
be/src/vec/functions/function_ignore.cpp:
##########
@@ -0,0 +1,50 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "vec/columns/columns_number.h"
+#include "vec/data_types/data_type_number.h"
+#include "vec/functions/function.h"
+#include "vec/functions/simple_function_factory.h"
+
+namespace doris::vectorized {
+
+class FunctionIgnore : public IFunction {
+public:
+    static constexpr auto name = "ignore";
+    String get_name() const override { return name; }
+
+    static FunctionPtr create() { return std::make_shared<FunctionIgnore>(); }
+    DataTypePtr get_return_type_impl(const DataTypes& arguments) const override {
+        return std::make_shared<DataTypeUInt8>();
+    }
+
+    bool is_variadic() const override { return true; }
+    size_t get_number_of_arguments() const override { return 0; }
+    bool use_default_implementation_for_nulls() const override { return false; }
+
+    Status execute_impl(FunctionContext* context, Block& block, const ColumnNumbers& arguments,
+                        size_t result, size_t input_rows_count) const override {

Review Comment:
   warning: method 'execute_impl' can be made static [readability-convert-member-functions-to-static]
   
   ```suggestion
       static Status execute_impl(FunctionContext* context, Block& block, const ColumnNumbers& arguments,
                           size_t result, size_t input_rows_count) override {
   ```
   



##########
be/src/vec/functions/function_ignore.cpp:
##########
@@ -0,0 +1,50 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "vec/columns/columns_number.h"
+#include "vec/data_types/data_type_number.h"
+#include "vec/functions/function.h"
+#include "vec/functions/simple_function_factory.h"
+
+namespace doris::vectorized {
+
+class FunctionIgnore : public IFunction {
+public:
+    static constexpr auto name = "ignore";
+    String get_name() const override { return name; }
+
+    static FunctionPtr create() { return std::make_shared<FunctionIgnore>(); }
+    DataTypePtr get_return_type_impl(const DataTypes& arguments) const override {

Review Comment:
   warning: method 'get_return_type_impl' can be made static [readability-convert-member-functions-to-static]
   
   ```suggestion
       static DataTypePtr get_return_type_impl(const DataTypes& arguments) override {
   ```
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Feature](functions) support ignore and nullable functions [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27848:
URL: https://github.com/apache/doris/pull/27848#issuecomment-1835372344

   
   <details>
   <summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
   
   ```
   Tpch sf100 test result on commit 20e1841edfe46ac49f40b3a8e621ffbd2d48ceca, data reload: false
   
   run tpch-sf100 query with default conf and session variables
   q1	4896	4655	4654	4654
   q2	360	156	156	156
   q3	1506	1267	1318	1267
   q4	1146	950	931	931
   q5	3228	3227	3222	3222
   q6	252	127	128	127
   q7	1021	519	532	519
   q8	2255	2252	2234	2234
   q9	6950	6978	6979	6978
   q10	3288	3349	3357	3349
   q11	346	204	212	204
   q12	348	223	217	217
   q13	4627	3931	4527	3931
   q14	249	223	221	221
   q15	595	536	535	535
   q16	442	409	400	400
   q17	1026	675	603	603
   q18	8228	7420	7370	7370
   q19	1561	1567	1533	1533
   q20	568	294	299	294
   q21	3415	2950	2965	2950
   q22	368	295	305	295
   Total cold run time: 46675 ms
   Total hot run time: 41990 ms
   
   run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
   q1	4579	4611	4594	4594
   q2	335	199	249	199
   q3	3752	3744	3731	3731
   q4	2546	2523	2520	2520
   q5	6211	6234	6194	6194
   q6	241	123	123	123
   q7	2613	2006	1990	1990
   q8	3741	3726	3765	3726
   q9	9440	9397	9438	9397
   q10	4072	4145	4147	4145
   q11	668	504	492	492
   q12	823	622	634	622
   q13	4370	3638	3643	3638
   q14	271	258	254	254
   q15	607	518	530	518
   q16	512	468	478	468
   q17	2144	2071	2121	2071
   q18	9649	8981	8996	8981
   q19	1795	1785	1772	1772
   q20	2312	1976	1967	1967
   q21	7315	6832	6905	6832
   q22	681	535	575	535
   Total cold run time: 68677 ms
   Total hot run time: 64769 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Feature](functions) support ignore and nullable functions [doris]

Posted by "zclllyybb (via GitHub)" <gi...@apache.org>.
zclllyybb commented on PR #27848:
URL: https://github.com/apache/doris/pull/27848#issuecomment-1835209530

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Feature](functions) support ignore and nullable functions [doris]

Posted by "zclllyybb (via GitHub)" <gi...@apache.org>.
zclllyybb commented on PR #27848:
URL: https://github.com/apache/doris/pull/27848#issuecomment-1837917161

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Feature](functions) support ignore and nullable functions [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27848:
URL: https://github.com/apache/doris/pull/27848#issuecomment-1840072299

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Feature](functions) support ignore and nullable functions [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27848:
URL: https://github.com/apache/doris/pull/27848#issuecomment-1837738391

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Feature](functions) support ignore and nullable functions [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27848:
URL: https://github.com/apache/doris/pull/27848#issuecomment-1837738421

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Feature](functions) support ignore and nullable functions [doris]

Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt merged PR #27848:
URL: https://github.com/apache/doris/pull/27848


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org