You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2020/04/17 10:25:22 UTC
[GitHub] [incubator-doris] yangzhg opened a new pull request #3345: [FIX]
fix doris string functions not support multibyte string
yangzhg opened a new pull request #3345: [FIX] fix doris string functions not support multibyte string
URL: https://github.com/apache/incubator-doris/pull/3345
fixes the bug describe in #3332 , doris string functions not support utf8 strings
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #3345:
[FIX] fix doris string functions not support multibyte string
Posted by GitBox <gi...@apache.org>.
imay commented on a change in pull request #3345: [FIX] fix doris string functions not support multibyte string
URL: https://github.com/apache/incubator-doris/pull/3345#discussion_r410144966
##########
File path: be/src/exprs/string_functions.cpp
##########
@@ -223,6 +259,36 @@ IntVal StringFunctions::length(FunctionContext* context, const StringVal& str) {
return IntVal(str.len);
}
+// Implementation of CHAR_LENGTH
+// int char_length(string input)
+// Returns the length of charactors of input. If input == NULL, returns
+// NULL per MySQL
+IntVal StringFunctions::char_length(FunctionContext* context, const StringVal& str) {
Review comment:
Is this only valid for UTF8 encode?
If it is, better to rename `utf8_length`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request
#3345: [FIX] fix doris string functions not support multibyte string
Posted by GitBox <gi...@apache.org>.
morningman commented on a change in pull request #3345: [FIX] fix doris string functions not support multibyte string
URL: https://github.com/apache/incubator-doris/pull/3345#discussion_r410261854
##########
File path: be/src/exprs/string_functions.cpp
##########
@@ -37,19 +37,55 @@ void StringFunctions::init() {
// - supported negative positions (count from the end of the string)
// - [optional] len. No len indicates longest substr possible
StringVal StringFunctions::substring(
- FunctionContext* context, const StringVal& str,
+ FunctionContext* context, const StringVal& str,
const IntVal& pos, const IntVal& len) {
if (str.is_null || pos.is_null || len.is_null) {
return StringVal::null();
}
+ if (len.val <= 0 || str.len == 0) {
+ return StringVal();
+ }
+
+ // create index indicate every char start byte
+ // e.g. "hello word 你好" => [0,1,2,3,4,5,6,7,8,9,10,11,14] 你 and 好 are 3 bytes
+ // why use a vector as index? if theres is no negative pos val it is unnecessary,
+ // but if has pos is negative it is not easy to determin where to start, so need a
+ // index save ervery charactor's length
Review comment:
```suggestion
// index save every character's length
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #3345:
[FIX] fix doris string functions not support multibyte string
Posted by GitBox <gi...@apache.org>.
imay commented on a change in pull request #3345: [FIX] fix doris string functions not support multibyte string
URL: https://github.com/apache/incubator-doris/pull/3345#discussion_r410147852
##########
File path: be/src/exprs/string_functions.cpp
##########
@@ -266,7 +332,25 @@ StringVal StringFunctions::reverse(FunctionContext* context, const StringVal& st
if (UNLIKELY(result.is_null)) {
return result;
}
- std::reverse_copy(str.ptr, str.ptr + str.len, result.ptr);
+
+ for (size_t i = 0, char_size = 0; i < str.len; i += char_size) {
+ unsigned char byte = (unsigned)(str.ptr)[i];
+ if (byte >= 0xFC) {
+ char_size = 6;
+ } else if (byte >= 0xF8) {
+ char_size = 5;
+ } else if (byte >= 0xF0) {
+ char_size = 4;
+ } else if (byte >= 0xE0) {
+ char_size = 3;
+ } else if (byte >= 0xC0) {
+ char_size = 2;
+ } else {
+ char_size = 1;
+ }
Review comment:
Can you write a function and to reuse it?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #3345: [FIX] fix doris string functions not support multibyte string
Posted by GitBox <gi...@apache.org>.
imay commented on a change in pull request #3345:
URL: https://github.com/apache/incubator-doris/pull/3345#discussion_r421205423
##########
File path: gensrc/script/doris_builtins_functions.py
##########
@@ -533,6 +533,8 @@
'15FunctionContextERKNS1_9StringValERKNS1_6IntValES6_'],
[['length'], 'INT', ['VARCHAR'],
'_ZN5doris15StringFunctions6lengthEPN9doris_udf15FunctionContextERKNS1_9StringValE'],
+ [['char_length'], 'INT', ['VARCHAR'],
Review comment:
Maybe you forget to add this file
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #3345: [FIX] fix doris string functions not support multibyte string
Posted by GitBox <gi...@apache.org>.
imay commented on a change in pull request #3345:
URL: https://github.com/apache/incubator-doris/pull/3345#discussion_r416355968
##########
File path: gensrc/script/doris_builtins_functions.py
##########
@@ -533,6 +533,8 @@
'15FunctionContextERKNS1_9StringValERKNS1_6IntValES6_'],
[['length'], 'INT', ['VARCHAR'],
'_ZN5doris15StringFunctions6lengthEPN9doris_udf15FunctionContextERKNS1_9StringValE'],
+ [['char_length'], 'INT', ['VARCHAR'],
Review comment:
Better to give a synonym CHARACTER_LENGTH, which is also supported in MySQL
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] yangzhg commented on pull request #3345: [FIX] fix doris string functions not support multibyte string
Posted by GitBox <gi...@apache.org>.
yangzhg commented on pull request #3345:
URL: https://github.com/apache/incubator-doris/pull/3345#issuecomment-624412771
> @yangzhg
> BE build failed, can you fix it?
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on pull request #3345: [FIX] fix doris string functions not support multibyte string
Posted by GitBox <gi...@apache.org>.
morningman commented on pull request #3345:
URL: https://github.com/apache/incubator-doris/pull/3345#issuecomment-619040404
@imay Hi，Looking for your review
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #3345: [FIX] fix doris string functions not support multibyte string
Posted by GitBox <gi...@apache.org>.
imay commented on a change in pull request #3345:
URL: https://github.com/apache/incubator-doris/pull/3345#discussion_r420689803
##########
File path: gensrc/script/doris_builtins_functions.py
##########
@@ -533,6 +533,8 @@
'15FunctionContextERKNS1_9StringValERKNS1_6IntValES6_'],
[['length'], 'INT', ['VARCHAR'],
'_ZN5doris15StringFunctions6lengthEPN9doris_udf15FunctionContextERKNS1_9StringValE'],
+ [['char_length'], 'INT', ['VARCHAR'],
Review comment:
```suggestion
[['char_length', 'character_length'], 'INT', ['VARCHAR'],
```
Prefer above type
##########
File path: be/src/exprs/string_functions.cpp
##########
@@ -260,13 +321,16 @@ StringVal StringFunctions::reverse(FunctionContext* context, const StringVal& st
return StringVal::null();
}
- // TODO pengyubing
- // StringVal result = StringVal::create_temp_string_val(context, str.len);
StringVal result(context, str.len);
if (UNLIKELY(result.is_null)) {
return result;
}
- std::reverse_copy(str.ptr, str.ptr + str.len, result.ptr);
+
+ for (size_t i = 0, char_size = 0; i < str.len; i += char_size) {
Review comment:
Should change function `reverse` documentation
##########
File path: docs/documentation/cn/sql-reference/sql-functions/string-functions/char_length.md
##########
@@ -0,0 +1,47 @@
+<!--
Review comment:
Should change this document according to current document format.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] wutiangan commented on a change in pull request #3345: [FIX] fix doris string functions not support multibyte string
Posted by GitBox <gi...@apache.org>.
wutiangan commented on a change in pull request #3345:
URL: https://github.com/apache/incubator-doris/pull/3345#discussion_r412283557
##########
File path: docs/documentation/cn/sql-reference/sql-functions/string-functions/right.md
##########
@@ -24,7 +24,7 @@ under the License.
`VARCHAR right(VARCHAR str)`
-它返回具有指定长度的字符串的右边部分
+它返回具有指定长度的字符串的右边部分, 度的单位为utf8字符
Review comment:
```suggestion
它返回具有指定长度的字符串的右边部分, 长度的单位为utf8字符
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] yangzhg commented on a change in pull request #3345: [FIX] fix doris string functions not support multibyte string
Posted by GitBox <gi...@apache.org>.
yangzhg commented on a change in pull request #3345:
URL: https://github.com/apache/incubator-doris/pull/3345#discussion_r415198500
##########
File path: gensrc/script/doris_builtins_functions.py
##########
@@ -533,6 +533,8 @@
'15FunctionContextERKNS1_9StringValERKNS1_6IntValES6_'],
[['length'], 'INT', ['VARCHAR'],
'_ZN5doris15StringFunctions6lengthEPN9doris_udf15FunctionContextERKNS1_9StringValE'],
+ [['char_length_utf8'], 'INT', ['VARCHAR'],
Review comment:
I think char_length_utf8 is better, it means `char length in utf8`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #3345: [FIX] fix doris string functions not support multibyte string
Posted by GitBox <gi...@apache.org>.
imay commented on a change in pull request #3345:
URL: https://github.com/apache/incubator-doris/pull/3345#discussion_r421244812
##########
File path: gensrc/script/doris_builtins_functions.py
##########
@@ -533,6 +533,10 @@
'15FunctionContextERKNS1_9StringValERKNS1_6IntValES6_'],
[['length'], 'INT', ['VARCHAR'],
'_ZN5doris15StringFunctions6lengthEPN9doris_udf15FunctionContextERKNS1_9StringValE'],
+ [['char_length'], 'INT', ['VARCHAR'],
+ '_ZN5doris15StringFunctions16char_utf8_lengthEPN9doris_udf15FunctionContextERKNS1_9StringValE'],
+ [['character_length'], 'INT', ['VARCHAR'],
+ '_ZN5doris15StringFunctions16char_utf8_lengthEPN9doris_udf15FunctionContextERKNS1_9StringValE'],
Review comment:
This can be merged in one line.
```suggestion
[['char_length', 'character_length'], 'INT', ['VARCHAR'],
'_ZN5doris15StringFunctions16char_utf8_lengthEPN9doris_udf15FunctionContextERKNS1_9StringValE'],
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] wutiangan commented on a change in pull request #3345: [FIX] fix doris string functions not support multibyte string
Posted by GitBox <gi...@apache.org>.
wutiangan commented on a change in pull request #3345:
URL: https://github.com/apache/incubator-doris/pull/3345#discussion_r412281633
##########
File path: docs/documentation/cn/sql-reference/sql-functions/string-functions/char_length_utf8.md
##########
@@ -0,0 +1,47 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# length
+## description
+### Syntax
+
+`INT char_length_utf8(VARCHAR str)`
+
+
+返回字符串的长度，对于多字节字符，返回的字符数。
Review comment:
```suggestion
返回字符串的长度，对于多字节字符，返回字符数。
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] yangzhg edited a comment on pull request #3345: [FIX] fix doris string functions not support multibyte string
Posted by GitBox <gi...@apache.org>.
yangzhg edited a comment on pull request #3345:
URL: https://github.com/apache/incubator-doris/pull/3345#issuecomment-624412771
> @yangzhg
> BE build failed, can you fix it?
@imay done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] yangzhg commented on pull request #3345: [FIX] fix doris string functions not support multibyte string
Posted by GitBox <gi...@apache.org>.
yangzhg commented on pull request #3345:
URL: https://github.com/apache/incubator-doris/pull/3345#issuecomment-624991376
@imay please review again
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] yangzhg commented on a change in pull request #3345: [FIX] fix doris string functions not support multibyte string
Posted by GitBox <gi...@apache.org>.
yangzhg commented on a change in pull request #3345:
URL: https://github.com/apache/incubator-doris/pull/3345#discussion_r413438461
##########
File path: run-ut.sh
##########
@@ -24,6 +24,7 @@ ROOT=`cd "$ROOT"; pwd`
export DORIS_HOME=${ROOT}
. ${DORIS_HOME}/env.sh
+export BUILD_TYPE=DEBUG
Review comment:
when runing unit test we need to enable DECHEK() to found more problems
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] wutiangan commented on a change in pull request #3345: [FIX] fix doris string functions not support multibyte string
Posted by GitBox <gi...@apache.org>.
wutiangan commented on a change in pull request #3345:
URL: https://github.com/apache/incubator-doris/pull/3345#discussion_r412282827
##########
File path: docs/documentation/cn/sql-reference/sql-functions/string-functions/length.md
##########
@@ -24,7 +24,7 @@ under the License.
`INT length(VARCHAR str)`
-返回字符串的长度，对于多字节字符，返回的字符数。比如5个两字节宽度字，返回的长度是10。
+返回字符串的字节。
Review comment:
返回字符串的字节数
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] kangkaisen commented on pull request #3345: [FIX] fix doris string functions not support multibyte string
Posted by GitBox <gi...@apache.org>.
kangkaisen commented on pull request #3345:
URL: https://github.com/apache/incubator-doris/pull/3345#issuecomment-624119517
@imay Please review again.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] imay commented on pull request #3345: [FIX] fix doris string functions not support multibyte string
Posted by GitBox <gi...@apache.org>.
imay commented on pull request #3345:
URL: https://github.com/apache/incubator-doris/pull/3345#issuecomment-624404460
> @imay Please review again.
You can assign this PR to yourself. Then you will be the moderator of this PR
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #3345: [FIX] fix doris string functions not support multibyte string
Posted by GitBox <gi...@apache.org>.
imay commented on a change in pull request #3345:
URL: https://github.com/apache/incubator-doris/pull/3345#discussion_r414679175
##########
File path: gensrc/script/doris_builtins_functions.py
##########
@@ -533,6 +533,8 @@
'15FunctionContextERKNS1_9StringValERKNS1_6IntValES6_'],
[['length'], 'INT', ['VARCHAR'],
'_ZN5doris15StringFunctions6lengthEPN9doris_udf15FunctionContextERKNS1_9StringValE'],
+ [['char_length_utf8'], 'INT', ['VARCHAR'],
Review comment:
why not utf8_length?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] imay commented on pull request #3345: [FIX] fix doris string functions not support multibyte string
Posted by GitBox <gi...@apache.org>.
imay commented on pull request #3345:
URL: https://github.com/apache/incubator-doris/pull/3345#issuecomment-624404025
@yangzhg
BE build failed, can you fix it?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request #3345: [FIX] fix doris string functions not support multibyte string
Posted by GitBox <gi...@apache.org>.
morningman commented on a change in pull request #3345:
URL: https://github.com/apache/incubator-doris/pull/3345#discussion_r412258949
##########
File path: docs/documentation/cn/sql-reference/sql-functions/string-functions/char_length_utf8.md
##########
@@ -0,0 +1,47 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# length
+## description
+### Syntax
+
+`INT char_length_utf8(VARCHAR str)`
+
+
+返回字符串的长度，对于多字节字符，返回的字符数。
+
+## example
+
+```
+mysql> select length("abc");
++---------------+
+| length('abc') |
++---------------+
+| 3 |
++---------------+
+
+mysql> select length("中国");
++------------------+
+| length('中国') |
++------------------+
+| 6 |
Review comment:
Should it be `2`?
##########
File path: be/src/exprs/string_functions.cpp
##########
@@ -19,37 +19,78 @@
#include <re2/re2.h>
-#include "exprs/expr.h"
#include "exprs/anyval_util.h"
+#include "exprs/expr.h"
+#include "math_functions.h"
#include "runtime/string_value.hpp"
#include "runtime/tuple_row.h"
#include "util/url_parser.h"
-#include "math_functions.h"
// NOTE: be careful not to use string::append. It is not performant.
namespace doris {
void StringFunctions::init() {
}
+size_t get_utf8_byte_length(unsigned char byte) {
+ size_t char_size = 0;
+ if (byte >= 0xFC) {
+ char_size = 6;
+ } else if (byte >= 0xF8) {
+ char_size = 5;
+ } else if (byte >= 0xF0) {
+ char_size = 4;
+ } else if (byte >= 0xE0) {
+ char_size = 3;
+ } else if (byte >= 0xC0) {
+ char_size = 2;
+ } else {
+ char_size = 1;
+ }
+ return char_size;
+}
+
// This behaves identically to the mysql implementation, namely:
// - 1-indexed positions
// - supported negative positions (count from the end of the string)
// - [optional] len. No len indicates longest substr possible
StringVal StringFunctions::substring(
- FunctionContext* context, const StringVal& str,
+ FunctionContext* context, const StringVal& str,
const IntVal& pos, const IntVal& len) {
if (str.is_null || pos.is_null || len.is_null) {
return StringVal::null();
}
+ if (len.val <= 0 || str.len == 0) {
+ return StringVal();
+ }
+
+ // create index indicate every char start byte
+ // e.g. "hello word 你好" => [0,1,2,3,4,5,6,7,8,9,10,11,14] 你 and 好 are 3 bytes
+ // why use a vector as index? It is unnecessary if there is no negative pos val,
+ // but if has pos is negative it is not easy to determin where to start, so need a
+ // index save every character's length
+ size_t byte_pos = 0;
+ std::vector<size_t> index;
+ for (size_t i = 0, char_size = 0; i < str.len; i += char_size) {
Review comment:
Can an optimization be done here?
1. Determine first, when `pos` is positive and greater than the length of str, return null directly, without for loop.
2. When `pos` is positive, can the for loop terminate after `pos + len` cycles? Instead of looping until the end of str?
##########
File path: docs/documentation/cn/sql-reference/sql-functions/string-functions/char_length_utf8.md
##########
@@ -0,0 +1,47 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# length
Review comment:
should it be `char_length_utf8`
##########
File path: run-ut.sh
##########
@@ -24,6 +24,7 @@ ROOT=`cd "$ROOT"; pwd`
export DORIS_HOME=${ROOT}
. ${DORIS_HOME}/env.sh
+export BUILD_TYPE=DEBUG
Review comment:
Why using this type?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] yangzhg commented on a change in pull request #3345: [FIX] fix doris string functions not support multibyte string
Posted by GitBox <gi...@apache.org>.
yangzhg commented on a change in pull request #3345:
URL: https://github.com/apache/incubator-doris/pull/3345#discussion_r421335854
##########
File path: gensrc/script/doris_builtins_functions.py
##########
@@ -533,6 +533,10 @@
'15FunctionContextERKNS1_9StringValERKNS1_6IntValES6_'],
[['length'], 'INT', ['VARCHAR'],
'_ZN5doris15StringFunctions6lengthEPN9doris_udf15FunctionContextERKNS1_9StringValE'],
+ [['char_length'], 'INT', ['VARCHAR'],
+ '_ZN5doris15StringFunctions16char_utf8_lengthEPN9doris_udf15FunctionContextERKNS1_9StringValE'],
+ [['character_length'], 'INT', ['VARCHAR'],
+ '_ZN5doris15StringFunctions16char_utf8_lengthEPN9doris_udf15FunctionContextERKNS1_9StringValE'],
Review comment:
done @imay
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] yangzhg edited a comment on pull request #3345: [FIX] fix doris string functions not support multibyte string
Posted by GitBox <gi...@apache.org>.
yangzhg edited a comment on pull request #3345:
URL: https://github.com/apache/incubator-doris/pull/3345#issuecomment-624412771
> @yangzhg
> BE build failed, can you fix it?
@imay `UT` has passed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request #3345: [FIX] fix doris string functions not support multibyte string
Posted by GitBox <gi...@apache.org>.
morningman commented on a change in pull request #3345:
URL: https://github.com/apache/incubator-doris/pull/3345#discussion_r416327527
##########
File path: gensrc/script/doris_builtins_functions.py
##########
@@ -533,6 +533,8 @@
'15FunctionContextERKNS1_9StringValERKNS1_6IntValES6_'],
[['length'], 'INT', ['VARCHAR'],
'_ZN5doris15StringFunctions6lengthEPN9doris_udf15FunctionContextERKNS1_9StringValE'],
+ [['char_length_utf8'], 'INT', ['VARCHAR'],
Review comment:
I found that in mysql , it calls `char_length()`,
https://dev.mysql.com/doc/refman/8.0/en/string-functions.html#function_char-length
So I think `char_length` is enough, and this function is unrelated with the encoding。
`char_length_utf8('中国')` and `char_length_gbk('中国')` should both return `2`.
So the suffix `_utf8` is unnecessary.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] yangzhg commented on a change in pull request #3345: [FIX] fix doris string functions not support multibyte string
Posted by GitBox <gi...@apache.org>.
yangzhg commented on a change in pull request #3345:
URL: https://github.com/apache/incubator-doris/pull/3345#discussion_r416354472
##########
File path: gensrc/script/doris_builtins_functions.py
##########
@@ -533,6 +533,8 @@
'15FunctionContextERKNS1_9StringValERKNS1_6IntValES6_'],
[['length'], 'INT', ['VARCHAR'],
'_ZN5doris15StringFunctions6lengthEPN9doris_udf15FunctionContextERKNS1_9StringValE'],
+ [['char_length_utf8'], 'INT', ['VARCHAR'],
Review comment:
@imay I have changed to char_length, please review it
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org