You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "hanyuzheng7 (via GitHub)" <gi...@apache.org> on 2023/07/31 20:28:34 UTC
[GitHub] [flink] hanyuzheng7 opened a new pull request, #23108: [FLINK-32706][table] Add built-in SPLIT_STRING function
hanyuzheng7 opened a new pull request, #23108:
URL: https://github.com/apache/flink/pull/23108
## What is the purpose of the change
This is an implementation of SPLIT
## Brief change log
Splits a string into an array of substrings based on a delimiter. If the delimiter is not found, then the original string is returned as the only element in the array. If the delimiter is empty, then all characters in the string are split. If either, string or delimiter, are NULL, then a NULL value is returned.
If the delimiter is found at the beginning or end of the string, or there are contiguous delimiters, then an empty space is added to the array.
- Syntax
`SPLIT(string, delimiter)`
- Arguments
string: The string need to split
delimiter: Splits a string into an array of substrings based on a delimiter
Returns
If the delimiter is not found, then the original string is returned as the only element in the array. If the delimiter is empty, then all characters in the string are split. If either, string or delimiter, are NULL, then a NULL value is returned.
- Examples
```
SELECT SPLIT('abcdefg', 'c');
Result: ['ab', 'defg']
```
- See also
1. ksqlDB Split function
ksqlDB provides a scalar function named SPLIT which splits a string into an array of substrings based on a delimiter.
Syntax: SPLIT(string, delimiter)
For example: SPLIT('a,b,c', ',') will return ['a', 'b', 'c'].
https://docs.ksqldb.io/en/0.8.1-ksqldb/developer-guide/ksqldb-reference/scalar-functions/#split
2. Apache Hive Split function
Hive offers a function named split which splits a string around a specified delimiter and returns an array of strings.
Syntax: array<string> split(string str, string pat)
For example: split('a,b,c', ',') will return ["a", "b", "c"].
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF
3. Spark SQL Split function
Spark SQL also offers a function named split, similar to the one in Hive.
Syntax: split(str, pattern[, limit])
Here, limit is an optional parameter to specify the maximum length of the returned array.
For example: split('oneAtwoBthreeC', '[ABC]', 2) will return ["one", "twoBthreeC"].
https://spark.apache.org/docs/latest/api/sql/index.html#split
4. Presto Split function
Presto offers a function named split which splits a string around a regular expression and returns an array of strings.
Syntax: array<varchar> split(string str, string regex)
For example: split('a.b.c', '\.') will return ["a", "b", "c"].
https://prestodb.io/docs/current/functions/string.html
## Verifying this change
This change added tests in CollectionFunctionsITCase.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (yes / no)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / no)
- The serializers: (yes / no / don't know)
- The runtime per-record code paths (performance sensitive): (yes / no / don't know)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know)
- The S3 file system connector: (yes / no / don't know)
## Documentation
- Does this pull request introduce a new feature? (yes / no)
- If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-32706][table] Add built-in SPLIT_STRING function [flink]
Posted by "hanyuzheng7 (via GitHub)" <gi...@apache.org>.
hanyuzheng7 closed pull request #23108: [FLINK-32706][table] Add built-in SPLIT_STRING function
URL: https://github.com/apache/flink/pull/23108
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] ferenc-csaky commented on a diff in pull request #23108: [FLINK-32706][table] Add built-in SPLIT_STRING function
Posted by "ferenc-csaky (via GitHub)" <gi...@apache.org>.
ferenc-csaky commented on code in PR #23108:
URL: https://github.com/apache/flink/pull/23108#discussion_r1288332845
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/scalar/SplitFunction.java:
##########
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions.scalar;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.data.ArrayData;
+import org.apache.flink.table.data.GenericArrayData;
+import org.apache.flink.table.data.StringData;
+import org.apache.flink.table.data.binary.BinaryStringData;
+import org.apache.flink.table.functions.BuiltInFunctionDefinitions;
+import org.apache.flink.table.functions.SpecializedFunction;
+import org.apache.flink.util.FlinkRuntimeException;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/** Implementation of {@link BuiltInFunctionDefinitions#SPLIT}. */
+@Internal
+public class SplitFunction extends BuiltInScalarFunction {
+ public SplitFunction(SpecializedFunction.SpecializedContext context) {
+ super(BuiltInFunctionDefinitions.SPLIT, context);
+ }
+
+ public @Nullable ArrayData eval(@Nullable StringData string, @Nullable StringData delimiter) {
+ try {
+ if (string == null || delimiter == null) {
+ return null;
+ }
+ String string1 = string.toString();
+ String delimiter1 = delimiter.toString();
+ List<StringData> resultList = new ArrayList<>();
+
+ if (delimiter1.equals("")) {
+ for (char c : string1.toCharArray()) {
+ resultList.add(BinaryStringData.fromString(String.valueOf(c)));
+ }
+ } else {
+ int start = 0;
+ int end = string1.indexOf(delimiter1);
+ while (end != -1) {
+ String substring = string1.substring(start, end);
+ resultList.add(
+ BinaryStringData.fromString(
+ substring.isEmpty()
+ ? ""
+ : substring)); // Added this check to handle consecutive
Review Comment:
Did not spot this the first time, but this check seems unnecessary. If `substring.isEmpty()` is true `substring` will be an empty string, so `BinaryStringData.fromString(substring)` will do the same thing.
The same is true for `remaining` @ line 72-74.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #23108: [FLINK-32706][table] Add built-in SPLIT_STRING function
Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #23108:
URL: https://github.com/apache/flink/pull/23108#issuecomment-1659125903
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0f6c18558aa89b3691698b5814fd24627849f7ee",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0f6c18558aa89b3691698b5814fd24627849f7ee",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0f6c18558aa89b3691698b5814fd24627849f7ee UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] ferenc-csaky commented on a diff in pull request #23108: [FLINK-32706][table] Add built-in SPLIT_STRING function
Posted by "ferenc-csaky (via GitHub)" <gi...@apache.org>.
ferenc-csaky commented on code in PR #23108:
URL: https://github.com/apache/flink/pull/23108#discussion_r1285662789
##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/CollectionFunctionsITCase.java:
##########
@@ -1193,4 +1194,70 @@ private Stream<TestSetSpec> arraySliceTestCases() {
" No match found for function signature ARRAY_SLICE()")
.testSqlValidationError("ARRAY_SLICE(null)", "Illegal use of 'NULL'"));
}
+
+ private Stream<TestSetSpec> splitTestCases() {
+ return Stream.of(
+ TestSetSpec.forFunction(BuiltInFunctionDefinitions.SPLIT)
+ .onFieldsWithData("123,123,23", null, ",123,123", ",123,123,", 123, "12345")
Review Comment:
I would use `,123,,123,` instead of `,123,123,` so there is a test case for contiguous delimiters as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] hanyuzheng7 commented on a diff in pull request #23108: [FLINK-32706][table] Add built-in SPLIT_STRING function
Posted by "hanyuzheng7 (via GitHub)" <gi...@apache.org>.
hanyuzheng7 commented on code in PR #23108:
URL: https://github.com/apache/flink/pull/23108#discussion_r1287697008
##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/CollectionFunctionsITCase.java:
##########
@@ -1193,4 +1194,70 @@ private Stream<TestSetSpec> arraySliceTestCases() {
" No match found for function signature ARRAY_SLICE()")
.testSqlValidationError("ARRAY_SLICE(null)", "Illegal use of 'NULL'"));
}
+
+ private Stream<TestSetSpec> splitTestCases() {
+ return Stream.of(
+ TestSetSpec.forFunction(BuiltInFunctionDefinitions.SPLIT)
+ .onFieldsWithData("123,123,23", null, ",123,123", ",123,123,", 123, "12345")
Review Comment:
Yes, you are right, I will add that test case, Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org