You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "liuyongvs (via GitHub)" <gi...@apache.org> on 2023/03/10 03:43:42 UTC
[GitHub] [flink] liuyongvs opened a new pull request, #22144: [FLINK-31102][table] Add ARRAY_REMOVE function.
liuyongvs opened a new pull request, #22144:
URL: https://github.com/apache/flink/pull/22144
- What is the purpose of the change
This is an implementation of ARRAY_REMOVE
- Brief change log
ARRAY_SIZE for Table API and SQL
```
Syntax:
array_remove(array)
Arguments:
array: An ARRAY to be handled.
Returns:
An ARRAY. If value is NULL, the result is NULL.
Examples:
SELECT array_remove(array(1, 2, 3, null, 3), 3);
-- [1,2,null]
```
See also
spark [[https://spark.apache.org/docs/latest/api/sql/index.html#array_size]](https://spark.apache.org/docs/latest/api/sql/index.html#array_remove)
presto https://prestodb.io/docs/current/functions/array.html
postgresql https://w3resource.com/PostgreSQL/postgresql_array_remove-function.php
- Verifying this change
This change added tests in CollectionFunctionsITCase
- Does this pull request potentially affect one of the following parts:
Dependencies (does it add or upgrade a dependency): ( no)
The public API, i.e., is any changed class annotated with @Public(Evolving): (yes )
The serializers: (no)
The runtime per-record code paths (performance sensitive): ( no)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: ( no)
The S3 file system connector: ( no)
- Documentation
Does this pull request introduce a new feature? (yes)
If yes, how is the feature documented? (docs)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] snuyanzin commented on a diff in pull request #22144: [FLINK-31102][table] Add ARRAY_REMOVE function.
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #22144:
URL: https://github.com/apache/flink/pull/22144#discussion_r1142753109
##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/CollectionFunctionsITCase.java:
##########
@@ -178,6 +181,125 @@ Stream<TestSetSpec> getTestSetSpecs() {
null
},
DataTypes.ARRAY(
- DataTypes.ROW(DataTypes.BOOLEAN(), DataTypes.DATE()))));
+ DataTypes.ROW(DataTypes.BOOLEAN(), DataTypes.DATE()))),
+ TestSetSpec.forFunction(BuiltInFunctionDefinitions.ARRAY_REMOVE)
+ .onFieldsWithData(
+ new Integer[] {1, 2, 2},
+ null,
+ new String[] {"Hello", "World"},
+ new Row[] {
+ Row.of(true, LocalDate.of(2022, 4, 20)),
+ Row.of(true, LocalDate.of(1990, 10, 14)),
+ null
+ },
+ new Integer[] {null, null, 1},
+ new Integer[][] {
+ new Integer[] {1, null, 3}, new Integer[] {0}, new Integer[] {1}
+ },
+ new Map[] {
+ CollectionUtil.map(entry(1, "a"), entry(2, "b")),
+ CollectionUtil.map(entry(3, "c"), entry(4, "d")),
+ null
+ })
+ .andDataTypes(
+ DataTypes.ARRAY(DataTypes.INT()),
+ DataTypes.ARRAY(DataTypes.INT()),
+ DataTypes.ARRAY(DataTypes.STRING()).notNull(),
+ DataTypes.ARRAY(
+ DataTypes.ROW(DataTypes.BOOLEAN(), DataTypes.DATE())),
+ DataTypes.ARRAY(DataTypes.INT()),
+ DataTypes.ARRAY(DataTypes.ARRAY(DataTypes.INT())),
+ DataTypes.ARRAY(DataTypes.MAP(DataTypes.INT(), DataTypes.STRING())))
+ // ARRAY<INT>
+ .testResult(
+ $("f0").arrayRemove(2),
+ "ARRAY_REMOVE(f0, 2)",
+ new Integer[] {1},
+ DataTypes.ARRAY(DataTypes.INT()).nullable())
+ .testResult(
+ $("f0").arrayRemove(42),
+ "ARRAY_REMOVE(f0, 42)",
+ new Integer[] {1, 2, 2},
+ DataTypes.ARRAY(DataTypes.INT()).nullable())
+ .testResult(
+ $("f0").arrayRemove(
+ lit(null, DataTypes.SMALLINT())
+ .cast(DataTypes.INT())),
+ "ARRAY_REMOVE(f0, cast(NULL AS INT))",
+ new Integer[] {1, 2, 2},
+ DataTypes.ARRAY(DataTypes.INT()).nullable())
+ // ARRAY<INT> of null value
+ .testResult(
+ $("f1").arrayRemove(12),
+ "ARRAY_REMOVE(f1, 12)",
+ null,
+ DataTypes.ARRAY(DataTypes.INT()).nullable())
+ .testResult(
+ $("f1").arrayRemove(null),
+ "ARRAY_REMOVE(f1, NULL)",
+ null,
+ DataTypes.ARRAY(DataTypes.INT()).nullable())
+ // ARRAY<STRING> NOT NULL
+ .testResult(
+ $("f2").arrayRemove("Hello"),
+ "ARRAY_REMOVE(f2, 'Hello')",
+ new String[] {"World"},
+ DataTypes.ARRAY(DataTypes.STRING()).notNull())
+ .testResult(
+ $("f2").arrayRemove(
+ lit(null, DataTypes.STRING())
+ .cast(DataTypes.STRING())),
+ "ARRAY_REMOVE(f2, cast(NULL AS VARCHAR))",
Review Comment:
i do not see a big difference between these tests and tests for int types... `ARRAY_REMOVE` invokes same code for both.... Probably just one (int or varchar is enough)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] liuyongvs commented on a diff in pull request #22144: [FLINK-31102][table] Add ARRAY_REMOVE function.
Posted by "liuyongvs (via GitHub)" <gi...@apache.org>.
liuyongvs commented on code in PR #22144:
URL: https://github.com/apache/flink/pull/22144#discussion_r1142850269
##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/CollectionFunctionsITCase.java:
##########
@@ -178,6 +181,125 @@ Stream<TestSetSpec> getTestSetSpecs() {
null
},
DataTypes.ARRAY(
- DataTypes.ROW(DataTypes.BOOLEAN(), DataTypes.DATE()))));
+ DataTypes.ROW(DataTypes.BOOLEAN(), DataTypes.DATE()))),
+ TestSetSpec.forFunction(BuiltInFunctionDefinitions.ARRAY_REMOVE)
+ .onFieldsWithData(
+ new Integer[] {1, 2, 2},
+ null,
+ new String[] {"Hello", "World"},
+ new Row[] {
+ Row.of(true, LocalDate.of(2022, 4, 20)),
+ Row.of(true, LocalDate.of(1990, 10, 14)),
+ null
+ },
+ new Integer[] {null, null, 1},
+ new Integer[][] {
+ new Integer[] {1, null, 3}, new Integer[] {0}, new Integer[] {1}
+ },
+ new Map[] {
+ CollectionUtil.map(entry(1, "a"), entry(2, "b")),
+ CollectionUtil.map(entry(3, "c"), entry(4, "d")),
+ null
+ })
+ .andDataTypes(
+ DataTypes.ARRAY(DataTypes.INT()),
+ DataTypes.ARRAY(DataTypes.INT()),
+ DataTypes.ARRAY(DataTypes.STRING()).notNull(),
+ DataTypes.ARRAY(
+ DataTypes.ROW(DataTypes.BOOLEAN(), DataTypes.DATE())),
+ DataTypes.ARRAY(DataTypes.INT()),
+ DataTypes.ARRAY(DataTypes.ARRAY(DataTypes.INT())),
+ DataTypes.ARRAY(DataTypes.MAP(DataTypes.INT(), DataTypes.STRING())))
+ // ARRAY<INT>
+ .testResult(
+ $("f0").arrayRemove(2),
+ "ARRAY_REMOVE(f0, 2)",
+ new Integer[] {1},
+ DataTypes.ARRAY(DataTypes.INT()).nullable())
+ .testResult(
+ $("f0").arrayRemove(42),
+ "ARRAY_REMOVE(f0, 42)",
+ new Integer[] {1, 2, 2},
+ DataTypes.ARRAY(DataTypes.INT()).nullable())
+ .testResult(
+ $("f0").arrayRemove(
+ lit(null, DataTypes.SMALLINT())
+ .cast(DataTypes.INT())),
+ "ARRAY_REMOVE(f0, cast(NULL AS INT))",
+ new Integer[] {1, 2, 2},
+ DataTypes.ARRAY(DataTypes.INT()).nullable())
+ // ARRAY<INT> of null value
+ .testResult(
+ $("f1").arrayRemove(12),
+ "ARRAY_REMOVE(f1, 12)",
+ null,
+ DataTypes.ARRAY(DataTypes.INT()).nullable())
+ .testResult(
+ $("f1").arrayRemove(null),
+ "ARRAY_REMOVE(f1, NULL)",
+ null,
+ DataTypes.ARRAY(DataTypes.INT()).nullable())
+ // ARRAY<STRING> NOT NULL
+ .testResult(
+ $("f2").arrayRemove("Hello"),
+ "ARRAY_REMOVE(f2, 'Hello')",
+ new String[] {"World"},
+ DataTypes.ARRAY(DataTypes.STRING()).notNull())
+ .testResult(
+ $("f2").arrayRemove(
+ lit(null, DataTypes.STRING())
+ .cast(DataTypes.STRING())),
+ "ARRAY_REMOVE(f2, cast(NULL AS VARCHAR))",
Review Comment:
ok, fixed, thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] liuyongvs commented on pull request #22144: [FLINK-31102][table] Add ARRAY_REMOVE function.
Posted by "liuyongvs (via GitHub)" <gi...@apache.org>.
liuyongvs commented on PR #22144:
URL: https://github.com/apache/flink/pull/22144#issuecomment-1477347441
hi @snuyanzin fix your reviews
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] snuyanzin commented on a diff in pull request #22144: [FLINK-31102][table] Add ARRAY_REMOVE function.
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #22144:
URL: https://github.com/apache/flink/pull/22144#discussion_r1134622026
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/scalar/ArrayRemoveFunction.java:
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions.scalar;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.data.ArrayData;
+import org.apache.flink.table.data.GenericArrayData;
+import org.apache.flink.table.functions.BuiltInFunctionDefinitions;
+import org.apache.flink.table.functions.FunctionContext;
+import org.apache.flink.table.functions.SpecializedFunction;
+import org.apache.flink.table.types.CollectionDataType;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.util.FlinkRuntimeException;
+
+import javax.annotation.Nullable;
+
+import java.lang.invoke.MethodHandle;
+import java.util.ArrayList;
+import java.util.List;
+
+import static org.apache.flink.table.api.Expressions.$;
+
+/** Implementation of {@link BuiltInFunctionDefinitions#ARRAY_REMOVE}. */
+@Internal
+public class ArrayRemoveFunction extends BuiltInScalarFunction {
+ private final ArrayData.ElementGetter elementGetter;
+ private final SpecializedFunction.ExpressionEvaluator equalityEvaluator;
+ private transient MethodHandle equalityHandle;
+
+ public ArrayRemoveFunction(SpecializedFunction.SpecializedContext context) {
+ super(BuiltInFunctionDefinitions.ARRAY_REMOVE, context);
+ final DataType elementDataType =
+ ((CollectionDataType) context.getCallContext().getArgumentDataTypes().get(0))
+ .getElementDataType();
+ final DataType needleDataType = context.getCallContext().getArgumentDataTypes().get(1);
+ elementGetter = ArrayData.createElementGetter(elementDataType.getLogicalType());
+ equalityEvaluator =
+ context.createEvaluator(
+ $("element").isEqual($("needle")),
+ DataTypes.BOOLEAN(),
+ DataTypes.FIELD("element", elementDataType.notNull().toInternal()),
+ DataTypes.FIELD("needle", needleDataType.notNull().toInternal()));
+ }
+
+ @Override
+ public void open(FunctionContext context) throws Exception {
+ equalityHandle = equalityEvaluator.open(context);
+ }
+
+ public @Nullable ArrayData eval(ArrayData haystack, Object needle) {
+ try {
+ if (haystack == null) {
+ return null;
+ }
+
+ List list = new ArrayList();
+ final int size = haystack.size();
+ for (int pos = 0; pos < size; pos++) {
+ final Object element = elementGetter.getElementOrNull(haystack, pos);
+ if ((element == null && needle != null)
+ || (element != null && needle == null)
+ || (element != null
+ && needle != null
+ && !(boolean) equalityHandle.invoke(element, needle))) {
Review Comment:
This could be simplified e.g. check for the negative case and continue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] snuyanzin commented on a diff in pull request #22144: [FLINK-31102][table] Add ARRAY_REMOVE function.
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #22144:
URL: https://github.com/apache/flink/pull/22144#discussion_r1139401943
##########
flink-python/pyflink/table/expression.py:
##########
@@ -1487,6 +1487,13 @@ def array_distinct(self) -> 'Expression':
"""
return _binary_op("arrayDistinct")(self)
+ def array_remove(self, needle) -> 'Expression':
+ """
+ Remove all elements that equal to element from array.
Review Comment:
```suggestion
Removes all elements that equal to element from array.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] snuyanzin commented on a diff in pull request #22144: [FLINK-31102][table] Add ARRAY_REMOVE function.
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #22144:
URL: https://github.com/apache/flink/pull/22144#discussion_r1142751406
##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/CollectionFunctionsITCase.java:
##########
@@ -178,6 +181,125 @@ Stream<TestSetSpec> getTestSetSpecs() {
null
},
DataTypes.ARRAY(
- DataTypes.ROW(DataTypes.BOOLEAN(), DataTypes.DATE()))));
+ DataTypes.ROW(DataTypes.BOOLEAN(), DataTypes.DATE()))),
+ TestSetSpec.forFunction(BuiltInFunctionDefinitions.ARRAY_REMOVE)
+ .onFieldsWithData(
+ new Integer[] {1, 2, 2},
+ null,
+ new String[] {"Hello", "World"},
+ new Row[] {
+ Row.of(true, LocalDate.of(2022, 4, 20)),
+ Row.of(true, LocalDate.of(1990, 10, 14)),
+ null
+ },
+ new Integer[] {null, null, 1},
+ new Integer[][] {
+ new Integer[] {1, null, 3}, new Integer[] {0}, new Integer[] {1}
+ },
+ new Map[] {
+ CollectionUtil.map(entry(1, "a"), entry(2, "b")),
+ CollectionUtil.map(entry(3, "c"), entry(4, "d")),
+ null
+ })
+ .andDataTypes(
+ DataTypes.ARRAY(DataTypes.INT()),
+ DataTypes.ARRAY(DataTypes.INT()),
+ DataTypes.ARRAY(DataTypes.STRING()).notNull(),
+ DataTypes.ARRAY(
+ DataTypes.ROW(DataTypes.BOOLEAN(), DataTypes.DATE())),
+ DataTypes.ARRAY(DataTypes.INT()),
+ DataTypes.ARRAY(DataTypes.ARRAY(DataTypes.INT())),
+ DataTypes.ARRAY(DataTypes.MAP(DataTypes.INT(), DataTypes.STRING())))
+ // ARRAY<INT>
+ .testResult(
+ $("f0").arrayRemove(2),
+ "ARRAY_REMOVE(f0, 2)",
+ new Integer[] {1},
+ DataTypes.ARRAY(DataTypes.INT()).nullable())
+ .testResult(
+ $("f0").arrayRemove(42),
+ "ARRAY_REMOVE(f0, 42)",
+ new Integer[] {1, 2, 2},
+ DataTypes.ARRAY(DataTypes.INT()).nullable())
+ .testResult(
+ $("f0").arrayRemove(
+ lit(null, DataTypes.SMALLINT())
+ .cast(DataTypes.INT())),
+ "ARRAY_REMOVE(f0, cast(NULL AS INT))",
+ new Integer[] {1, 2, 2},
+ DataTypes.ARRAY(DataTypes.INT()).nullable())
+ // ARRAY<INT> of null value
+ .testResult(
+ $("f1").arrayRemove(12),
+ "ARRAY_REMOVE(f1, 12)",
+ null,
+ DataTypes.ARRAY(DataTypes.INT()).nullable())
+ .testResult(
+ $("f1").arrayRemove(null),
+ "ARRAY_REMOVE(f1, NULL)",
+ null,
+ DataTypes.ARRAY(DataTypes.INT()).nullable())
+ // ARRAY<STRING> NOT NULL
+ .testResult(
+ $("f2").arrayRemove("Hello"),
+ "ARRAY_REMOVE(f2, 'Hello')",
+ new String[] {"World"},
+ DataTypes.ARRAY(DataTypes.STRING()).notNull())
+ .testResult(
+ $("f2").arrayRemove(
+ lit(null, DataTypes.STRING())
+ .cast(DataTypes.STRING())),
+ "ARRAY_REMOVE(f2, cast(NULL AS VARCHAR))",
+ new String[] {"Hello", "World"},
+ DataTypes.ARRAY(DataTypes.STRING()).notNull())
+ // ARRAY<ROW<BOOLEAN, DATE>>
+ .testResult(
+ $("f3").arrayRemove(row(true, LocalDate.of(1990, 10, 14))),
+ "ARRAY_REMOVE(f3, (TRUE, DATE '1990-10-14'))",
+ new Row[] {Row.of(true, LocalDate.of(2022, 4, 20)), null},
+ DataTypes.ARRAY(
+ DataTypes.ROW(
+ DataTypes.BOOLEAN(), DataTypes.DATE()))
+ .nullable())
+ .testResult(
+ $("f3").arrayRemove(null),
+ "ARRAY_REMOVE(f3, null)",
+ new Row[] {
+ Row.of(true, LocalDate.of(2022, 4, 20)),
+ Row.of(true, LocalDate.of(1990, 10, 14)),
+ },
+ DataTypes.ARRAY(
+ DataTypes.ROW(
+ DataTypes.BOOLEAN(), DataTypes.DATE()))
+ .nullable())
+ // ARRAY<INT> with null elements
+ .testResult(
+ $("f4").arrayRemove(null),
+ "ARRAY_REMOVE(f4, NULL)",
+ new Integer[] {1},
+ DataTypes.ARRAY(DataTypes.INT()).nullable())
+ // ARRAY<ARRAY<INT>>
+ .testResult(
+ $("f5").arrayRemove(new Integer[] {0}),
+ "ARRAY_REMOVE(f5, array[0])",
+ new Integer[][] {new Integer[] {1, null, 3}, new Integer[] {1}},
+ DataTypes.ARRAY(DataTypes.ARRAY(DataTypes.INT()).nullable()))
+ // ARRAY<Map<INT, STRING>> with null elements
Review Comment:
```suggestion
// ARRAY<MAP<INT, STRING>> with null elements
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] snuyanzin commented on pull request #22144: [FLINK-31102][table] Add ARRAY_REMOVE function.
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on PR #22144:
URL: https://github.com/apache/flink/pull/22144#issuecomment-1463428022
thanks for the contribution
can you please fix description?
e.g.
1. I didn't get how `ARRAY_SIZE`is related to this PR
2. >Syntax:
array_remove(array)
is it possible to invoke `array_remove` for one arg? What is the expected output?
3. >SELECT array_remove(array(1, 2, 3, null, 3), 3);
it will not work in Flink since it is not Flink syntax
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] liuyongvs commented on pull request #22144: [FLINK-31102][table] Add ARRAY_REMOVE function.
Posted by "liuyongvs (via GitHub)" <gi...@apache.org>.
liuyongvs commented on PR #22144:
URL: https://github.com/apache/flink/pull/22144#issuecomment-1463440425
@snuyanzin
1.it is here after talking in https://github.com/apache/flink/pull/21947, which you reviewed before.
2. is it possible to invoke array_remove for one arg -> no possible. sorry. it is my misspelling。
3. it is spark/presto/postgre syntax. i think we can support it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] snuyanzin commented on a diff in pull request #22144: [FLINK-31102][table] Add ARRAY_REMOVE function.
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #22144:
URL: https://github.com/apache/flink/pull/22144#discussion_r1142750315
##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/CollectionFunctionsITCase.java:
##########
@@ -178,6 +181,125 @@ Stream<TestSetSpec> getTestSetSpecs() {
null
},
DataTypes.ARRAY(
- DataTypes.ROW(DataTypes.BOOLEAN(), DataTypes.DATE()))));
+ DataTypes.ROW(DataTypes.BOOLEAN(), DataTypes.DATE()))),
+ TestSetSpec.forFunction(BuiltInFunctionDefinitions.ARRAY_REMOVE)
+ .onFieldsWithData(
+ new Integer[] {1, 2, 2},
+ null,
+ new String[] {"Hello", "World"},
+ new Row[] {
+ Row.of(true, LocalDate.of(2022, 4, 20)),
+ Row.of(true, LocalDate.of(1990, 10, 14)),
+ null
+ },
+ new Integer[] {null, null, 1},
+ new Integer[][] {
+ new Integer[] {1, null, 3}, new Integer[] {0}, new Integer[] {1}
+ },
+ new Map[] {
+ CollectionUtil.map(entry(1, "a"), entry(2, "b")),
+ CollectionUtil.map(entry(3, "c"), entry(4, "d")),
+ null
+ })
+ .andDataTypes(
+ DataTypes.ARRAY(DataTypes.INT()),
+ DataTypes.ARRAY(DataTypes.INT()),
+ DataTypes.ARRAY(DataTypes.STRING()).notNull(),
+ DataTypes.ARRAY(
+ DataTypes.ROW(DataTypes.BOOLEAN(), DataTypes.DATE())),
+ DataTypes.ARRAY(DataTypes.INT()),
+ DataTypes.ARRAY(DataTypes.ARRAY(DataTypes.INT())),
+ DataTypes.ARRAY(DataTypes.MAP(DataTypes.INT(), DataTypes.STRING())))
+ // ARRAY<INT>
+ .testResult(
+ $("f0").arrayRemove(2),
+ "ARRAY_REMOVE(f0, 2)",
+ new Integer[] {1},
+ DataTypes.ARRAY(DataTypes.INT()).nullable())
+ .testResult(
+ $("f0").arrayRemove(42),
+ "ARRAY_REMOVE(f0, 42)",
+ new Integer[] {1, 2, 2},
+ DataTypes.ARRAY(DataTypes.INT()).nullable())
+ .testResult(
+ $("f0").arrayRemove(
+ lit(null, DataTypes.SMALLINT())
+ .cast(DataTypes.INT())),
+ "ARRAY_REMOVE(f0, cast(NULL AS INT))",
Review Comment:
```suggestion
"ARRAY_REMOVE(f0, CAST(NULL AS INT))",
```
All SQL keywords should be in uppercase
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] liuyongvs commented on pull request #22144: [FLINK-31102][table] Add ARRAY_REMOVE function.
Posted by "liuyongvs (via GitHub)" <gi...@apache.org>.
liuyongvs commented on PR #22144:
URL: https://github.com/apache/flink/pull/22144#issuecomment-1479002669
hi @snuyanzin fix your reviews
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] liuyongvs commented on pull request #22144: [FLINK-31102][table] Add ARRAY_REMOVE function.
Posted by "liuyongvs (via GitHub)" <gi...@apache.org>.
liuyongvs commented on PR #22144:
URL: https://github.com/apache/flink/pull/22144#issuecomment-1466116542
hi @snuyanzin could you have time to help review it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] snuyanzin commented on a diff in pull request #22144: [FLINK-31102][table] Add ARRAY_REMOVE function.
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #22144:
URL: https://github.com/apache/flink/pull/22144#discussion_r1134628496
##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/CollectionFunctionsITCase.java:
##########
@@ -178,6 +181,137 @@ Stream<TestSetSpec> getTestSetSpecs() {
null
},
DataTypes.ARRAY(
- DataTypes.ROW(DataTypes.BOOLEAN(), DataTypes.DATE()))));
+ DataTypes.ROW(DataTypes.BOOLEAN(), DataTypes.DATE()))),
+ TestSetSpec.forFunction(BuiltInFunctionDefinitions.ARRAY_REMOVE)
+ .onFieldsWithData(
+ new Integer[] {1, 2, 2},
+ null,
+ new String[] {"Hello", "World"},
+ new Row[] {
+ Row.of(true, LocalDate.of(2022, 4, 20)),
+ Row.of(true, LocalDate.of(1990, 10, 14)),
+ null
+ },
+ new Integer[] {null, null, 1},
+ new Integer[][] {
+ new Integer[] {1, null, 3}, new Integer[] {0}, new Integer[] {1}
+ },
+ new Map[] {
+ CollectionUtil.map(entry(1, "a"), entry(2, "b")),
+ CollectionUtil.map(entry(3, "c"), entry(4, "d")),
+ null
+ })
+ .andDataTypes(
+ DataTypes.ARRAY(DataTypes.INT()),
+ DataTypes.ARRAY(DataTypes.INT()),
+ DataTypes.ARRAY(DataTypes.STRING()).notNull(),
+ DataTypes.ARRAY(
+ DataTypes.ROW(DataTypes.BOOLEAN(), DataTypes.DATE())),
+ DataTypes.ARRAY(DataTypes.INT()),
+ DataTypes.ARRAY(DataTypes.ARRAY(DataTypes.INT())),
+ DataTypes.ARRAY(DataTypes.MAP(DataTypes.INT(), DataTypes.STRING())))
+ // ARRAY<INT>
+ .testResult(
+ $("f0").arrayRemove(2),
+ "ARRAY_REMOVE(f0, 2)",
+ new Integer[] {1},
+ DataTypes.ARRAY(DataTypes.INT()).nullable())
+ .testResult(
+ $("f0").arrayRemove(42),
+ "ARRAY_REMOVE(f0, 42)",
+ new Integer[] {1, 2, 2},
+ DataTypes.ARRAY(DataTypes.INT()).nullable())
+ .testResult(
+ $("f0").arrayRemove(
+ lit(null, DataTypes.SMALLINT())
+ .cast(DataTypes.INT())),
+ "ARRAY_REMOVE(f0, cast(NULL AS INT))",
+ new Integer[] {1, 2, 2},
+ DataTypes.ARRAY(DataTypes.INT()).nullable())
+ // ARRAY<INT> of null value
+ .testResult(
+ $("f1").arrayRemove(12),
+ "ARRAY_REMOVE(f1, 12)",
+ null,
+ DataTypes.ARRAY(DataTypes.INT()).nullable())
+ .testResult(
+ $("f1").arrayRemove(null),
+ "ARRAY_REMOVE(f1, NULL)",
+ null,
+ DataTypes.ARRAY(DataTypes.INT()).nullable())
+ // ARRAY<STRING> NOT NULL
+ .testResult(
+ $("f2").arrayRemove("Hello"),
+ "ARRAY_REMOVE(f2, 'Hello')",
+ new String[] {"World"},
+ DataTypes.ARRAY(DataTypes.STRING()).notNull())
+ .testResult(
+ $("f2").arrayRemove(
+ lit(null, DataTypes.STRING())
+ .cast(DataTypes.STRING())),
+ "ARRAY_REMOVE(f2, cast(NULL AS VARCHAR))",
+ new String[] {"Hello", "World"},
+ DataTypes.ARRAY(DataTypes.STRING()).notNull())
+ // ARRAY<ROW<BOOLEAN, DATE>>
+ .testResult(
+ $("f3").arrayRemove(row(true, LocalDate.of(1990, 10, 14))),
+ "ARRAY_REMOVE(f3, (TRUE, DATE '1990-10-14'))",
+ new Row[] {Row.of(true, LocalDate.of(2022, 4, 20)), null},
+ DataTypes.ARRAY(
+ DataTypes.ROW(
+ DataTypes.BOOLEAN(), DataTypes.DATE()))
+ .nullable())
+ .testResult(
+ $("f3").arrayRemove(row(false, LocalDate.of(1990, 10, 14))),
+ "ARRAY_REMOVE(f3, (FALSE, DATE '1990-10-14'))",
+ new Row[] {
+ Row.of(true, LocalDate.of(2022, 4, 20)),
+ Row.of(true, LocalDate.of(1990, 10, 14)),
+ null
+ },
+ DataTypes.ARRAY(
+ DataTypes.ROW(
+ DataTypes.BOOLEAN(), DataTypes.DATE()))
+ .nullable())
Review Comment:
Under the hood there are a lot of Flink jobs executed.
From one side testing is important but tests should be well chosen.
It seems e.g. one of this could be removed since they are very similar.
I guess up to 4-5 test cases should be enough.
WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] liuyongvs commented on a diff in pull request #22144: [FLINK-31102][table] Add ARRAY_REMOVE function.
Posted by "liuyongvs (via GitHub)" <gi...@apache.org>.
liuyongvs commented on code in PR #22144:
URL: https://github.com/apache/flink/pull/22144#discussion_r1135000970
##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/CollectionFunctionsITCase.java:
##########
@@ -178,6 +181,137 @@ Stream<TestSetSpec> getTestSetSpecs() {
null
},
DataTypes.ARRAY(
- DataTypes.ROW(DataTypes.BOOLEAN(), DataTypes.DATE()))));
+ DataTypes.ROW(DataTypes.BOOLEAN(), DataTypes.DATE()))),
+ TestSetSpec.forFunction(BuiltInFunctionDefinitions.ARRAY_REMOVE)
+ .onFieldsWithData(
+ new Integer[] {1, 2, 2},
+ null,
+ new String[] {"Hello", "World"},
+ new Row[] {
+ Row.of(true, LocalDate.of(2022, 4, 20)),
+ Row.of(true, LocalDate.of(1990, 10, 14)),
+ null
+ },
+ new Integer[] {null, null, 1},
+ new Integer[][] {
+ new Integer[] {1, null, 3}, new Integer[] {0}, new Integer[] {1}
+ },
+ new Map[] {
+ CollectionUtil.map(entry(1, "a"), entry(2, "b")),
+ CollectionUtil.map(entry(3, "c"), entry(4, "d")),
+ null
+ })
+ .andDataTypes(
+ DataTypes.ARRAY(DataTypes.INT()),
+ DataTypes.ARRAY(DataTypes.INT()),
+ DataTypes.ARRAY(DataTypes.STRING()).notNull(),
+ DataTypes.ARRAY(
+ DataTypes.ROW(DataTypes.BOOLEAN(), DataTypes.DATE())),
+ DataTypes.ARRAY(DataTypes.INT()),
+ DataTypes.ARRAY(DataTypes.ARRAY(DataTypes.INT())),
+ DataTypes.ARRAY(DataTypes.MAP(DataTypes.INT(), DataTypes.STRING())))
+ // ARRAY<INT>
+ .testResult(
+ $("f0").arrayRemove(2),
+ "ARRAY_REMOVE(f0, 2)",
+ new Integer[] {1},
+ DataTypes.ARRAY(DataTypes.INT()).nullable())
+ .testResult(
+ $("f0").arrayRemove(42),
+ "ARRAY_REMOVE(f0, 42)",
+ new Integer[] {1, 2, 2},
+ DataTypes.ARRAY(DataTypes.INT()).nullable())
+ .testResult(
+ $("f0").arrayRemove(
+ lit(null, DataTypes.SMALLINT())
+ .cast(DataTypes.INT())),
+ "ARRAY_REMOVE(f0, cast(NULL AS INT))",
+ new Integer[] {1, 2, 2},
+ DataTypes.ARRAY(DataTypes.INT()).nullable())
+ // ARRAY<INT> of null value
+ .testResult(
+ $("f1").arrayRemove(12),
+ "ARRAY_REMOVE(f1, 12)",
+ null,
+ DataTypes.ARRAY(DataTypes.INT()).nullable())
+ .testResult(
+ $("f1").arrayRemove(null),
+ "ARRAY_REMOVE(f1, NULL)",
+ null,
+ DataTypes.ARRAY(DataTypes.INT()).nullable())
+ // ARRAY<STRING> NOT NULL
+ .testResult(
+ $("f2").arrayRemove("Hello"),
+ "ARRAY_REMOVE(f2, 'Hello')",
+ new String[] {"World"},
+ DataTypes.ARRAY(DataTypes.STRING()).notNull())
+ .testResult(
+ $("f2").arrayRemove(
+ lit(null, DataTypes.STRING())
+ .cast(DataTypes.STRING())),
+ "ARRAY_REMOVE(f2, cast(NULL AS VARCHAR))",
+ new String[] {"Hello", "World"},
+ DataTypes.ARRAY(DataTypes.STRING()).notNull())
+ // ARRAY<ROW<BOOLEAN, DATE>>
+ .testResult(
+ $("f3").arrayRemove(row(true, LocalDate.of(1990, 10, 14))),
+ "ARRAY_REMOVE(f3, (TRUE, DATE '1990-10-14'))",
+ new Row[] {Row.of(true, LocalDate.of(2022, 4, 20)), null},
+ DataTypes.ARRAY(
+ DataTypes.ROW(
+ DataTypes.BOOLEAN(), DataTypes.DATE()))
+ .nullable())
+ .testResult(
+ $("f3").arrayRemove(row(false, LocalDate.of(1990, 10, 14))),
+ "ARRAY_REMOVE(f3, (FALSE, DATE '1990-10-14'))",
+ new Row[] {
+ Row.of(true, LocalDate.of(2022, 4, 20)),
+ Row.of(true, LocalDate.of(1990, 10, 14)),
+ null
+ },
+ DataTypes.ARRAY(
+ DataTypes.ROW(
+ DataTypes.BOOLEAN(), DataTypes.DATE()))
+ .nullable())
Review Comment:
i agree with you @snuyanzin
These examples cover a variety of cases, and the rest of the functions don't have to cover as many cases as possible, but some of the early functions still need to cover as many cases as possible, so that users can see how to use them and find bugs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] liuyongvs commented on pull request #22144: [FLINK-31102][table] Add ARRAY_REMOVE function.
Posted by "liuyongvs (via GitHub)" <gi...@apache.org>.
liuyongvs commented on PR #22144:
URL: https://github.com/apache/flink/pull/22144#issuecomment-1463213377
hi @snuyanzin @twalthr after talking in https://github.com/apache/flink/pull/21947
1) do not support array_size, which can be replaced by CARDINALITY
2) the select array_remove(array[1, 2], cast(null as int)); throw exception, which is fixed and merge in https://github.com/apache/flink/commit/10dce7cf0a04b80d7416a5760e1a6dbc430d9f88
3) the java equals may cause bug in array_distinct, should use ExpressionEvaluator, suggested by @twalthr which is fixed and merge in https://github.com/apache/flink/commit/6797d6f2592373b2606ddd8c8aad316d677c1cc6
4) the last problem found in https://github.com/apache/flink/pull/22143#issuecomment-1463191208, will also need to be solved
so it is the time to implment other array functions. so i submit a new PR, could you help to review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #22144: [FLINK-31102][table] Add ARRAY_REMOVE function.
Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #22144:
URL: https://github.com/apache/flink/pull/22144#issuecomment-1463210357
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "603c5f92ab0c367e0ccc0ea753fa810a8da06b8f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "603c5f92ab0c367e0ccc0ea753fa810a8da06b8f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 603c5f92ab0c367e0ccc0ea753fa810a8da06b8f UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] liuyongvs commented on pull request #22144: [FLINK-31102][table] Add ARRAY_REMOVE function.
Posted by "liuyongvs (via GitHub)" <gi...@apache.org>.
liuyongvs commented on PR #22144:
URL: https://github.com/apache/flink/pull/22144#issuecomment-1468087609
@flinkbot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] liuyongvs commented on pull request #22144: [FLINK-31102][table] Add ARRAY_REMOVE function.
Posted by "liuyongvs (via GitHub)" <gi...@apache.org>.
liuyongvs commented on PR #22144:
URL: https://github.com/apache/flink/pull/22144#issuecomment-1475499350
hi @snuyanzin could you help review it again?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] liuyongvs commented on pull request #22144: [FLINK-31102][table] Add ARRAY_REMOVE function.
Posted by "liuyongvs (via GitHub)" <gi...@apache.org>.
liuyongvs commented on PR #22144:
URL: https://github.com/apache/flink/pull/22144#issuecomment-1483680915
hi @snuyanzin , will it be merged now?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] liuyongvs commented on a diff in pull request #22144: [FLINK-31102][table] Add ARRAY_REMOVE function.
Posted by "liuyongvs (via GitHub)" <gi...@apache.org>.
liuyongvs commented on code in PR #22144:
URL: https://github.com/apache/flink/pull/22144#discussion_r1134999542
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/scalar/ArrayRemoveFunction.java:
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions.scalar;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.data.ArrayData;
+import org.apache.flink.table.data.GenericArrayData;
+import org.apache.flink.table.functions.BuiltInFunctionDefinitions;
+import org.apache.flink.table.functions.FunctionContext;
+import org.apache.flink.table.functions.SpecializedFunction;
+import org.apache.flink.table.types.CollectionDataType;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.util.FlinkRuntimeException;
+
+import javax.annotation.Nullable;
+
+import java.lang.invoke.MethodHandle;
+import java.util.ArrayList;
+import java.util.List;
+
+import static org.apache.flink.table.api.Expressions.$;
+
+/** Implementation of {@link BuiltInFunctionDefinitions#ARRAY_REMOVE}. */
+@Internal
+public class ArrayRemoveFunction extends BuiltInScalarFunction {
+ private final ArrayData.ElementGetter elementGetter;
+ private final SpecializedFunction.ExpressionEvaluator equalityEvaluator;
+ private transient MethodHandle equalityHandle;
+
+ public ArrayRemoveFunction(SpecializedFunction.SpecializedContext context) {
+ super(BuiltInFunctionDefinitions.ARRAY_REMOVE, context);
+ final DataType elementDataType =
+ ((CollectionDataType) context.getCallContext().getArgumentDataTypes().get(0))
+ .getElementDataType();
+ final DataType needleDataType = context.getCallContext().getArgumentDataTypes().get(1);
+ elementGetter = ArrayData.createElementGetter(elementDataType.getLogicalType());
+ equalityEvaluator =
+ context.createEvaluator(
+ $("element").isEqual($("needle")),
+ DataTypes.BOOLEAN(),
+ DataTypes.FIELD("element", elementDataType.notNull().toInternal()),
+ DataTypes.FIELD("needle", needleDataType.notNull().toInternal()));
+ }
+
+ @Override
+ public void open(FunctionContext context) throws Exception {
+ equalityHandle = equalityEvaluator.open(context);
+ }
+
+ public @Nullable ArrayData eval(ArrayData haystack, Object needle) {
+ try {
+ if (haystack == null) {
+ return null;
+ }
+
+ List list = new ArrayList();
+ final int size = haystack.size();
+ for (int pos = 0; pos < size; pos++) {
+ final Object element = elementGetter.getElementOrNull(haystack, pos);
+ if ((element == null && needle != null)
+ || (element != null && needle == null)
+ || (element != null
+ && needle != null
+ && !(boolean) equalityHandle.invoke(element, needle))) {
Review Comment:
have done @snuyanzin
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] liuyongvs commented on a diff in pull request #22144: [FLINK-31102][table] Add ARRAY_REMOVE function.
Posted by "liuyongvs (via GitHub)" <gi...@apache.org>.
liuyongvs commented on code in PR #22144:
URL: https://github.com/apache/flink/pull/22144#discussion_r1139573434
##########
docs/data/sql_functions.yml:
##########
@@ -617,6 +617,9 @@ collection:
- sql: ARRAY_DISTINCT(haystack)
table: haystack.arrayDistinct()
description: Returns an array with unique elements. If the array itself is null, the function will return null. Keeps ordering of elements.
+ - sql: ARRAY_REMOVE(haystack, needle)
+ table: haystack.arrayRemove(needle)
+ description: Remove all elements that equal to element from array. If the array itself is null, the function will return null. Keeps ordering of elements.
Review Comment:
have fixed
##########
flink-python/pyflink/table/expression.py:
##########
@@ -1487,6 +1487,13 @@ def array_distinct(self) -> 'Expression':
"""
return _binary_op("arrayDistinct")(self)
+ def array_remove(self, needle) -> 'Expression':
+ """
+ Remove all elements that equal to element from array.
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] snuyanzin commented on a diff in pull request #22144: [FLINK-31102][table] Add ARRAY_REMOVE function.
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #22144:
URL: https://github.com/apache/flink/pull/22144#discussion_r1139402152
##########
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/internal/BaseExpressions.java:
##########
@@ -1359,6 +1360,16 @@ public OutType arrayDistinct() {
return toApiSpecificExpression(unresolvedCall(ARRAY_DISTINCT, toExpr()));
}
+ /**
+ * Remove all elements that equal to element from array.
Review Comment:
```suggestion
* Removes all elements that equal to element from array.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] snuyanzin commented on a diff in pull request #22144: [FLINK-31102][table] Add ARRAY_REMOVE function.
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #22144:
URL: https://github.com/apache/flink/pull/22144#discussion_r1142750566
##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/CollectionFunctionsITCase.java:
##########
@@ -178,6 +181,125 @@ Stream<TestSetSpec> getTestSetSpecs() {
null
},
DataTypes.ARRAY(
- DataTypes.ROW(DataTypes.BOOLEAN(), DataTypes.DATE()))));
+ DataTypes.ROW(DataTypes.BOOLEAN(), DataTypes.DATE()))),
+ TestSetSpec.forFunction(BuiltInFunctionDefinitions.ARRAY_REMOVE)
+ .onFieldsWithData(
+ new Integer[] {1, 2, 2},
+ null,
+ new String[] {"Hello", "World"},
+ new Row[] {
+ Row.of(true, LocalDate.of(2022, 4, 20)),
+ Row.of(true, LocalDate.of(1990, 10, 14)),
+ null
+ },
+ new Integer[] {null, null, 1},
+ new Integer[][] {
+ new Integer[] {1, null, 3}, new Integer[] {0}, new Integer[] {1}
+ },
+ new Map[] {
+ CollectionUtil.map(entry(1, "a"), entry(2, "b")),
+ CollectionUtil.map(entry(3, "c"), entry(4, "d")),
+ null
+ })
+ .andDataTypes(
+ DataTypes.ARRAY(DataTypes.INT()),
+ DataTypes.ARRAY(DataTypes.INT()),
+ DataTypes.ARRAY(DataTypes.STRING()).notNull(),
+ DataTypes.ARRAY(
+ DataTypes.ROW(DataTypes.BOOLEAN(), DataTypes.DATE())),
+ DataTypes.ARRAY(DataTypes.INT()),
+ DataTypes.ARRAY(DataTypes.ARRAY(DataTypes.INT())),
+ DataTypes.ARRAY(DataTypes.MAP(DataTypes.INT(), DataTypes.STRING())))
+ // ARRAY<INT>
+ .testResult(
+ $("f0").arrayRemove(2),
+ "ARRAY_REMOVE(f0, 2)",
+ new Integer[] {1},
+ DataTypes.ARRAY(DataTypes.INT()).nullable())
+ .testResult(
+ $("f0").arrayRemove(42),
+ "ARRAY_REMOVE(f0, 42)",
+ new Integer[] {1, 2, 2},
+ DataTypes.ARRAY(DataTypes.INT()).nullable())
+ .testResult(
+ $("f0").arrayRemove(
+ lit(null, DataTypes.SMALLINT())
+ .cast(DataTypes.INT())),
+ "ARRAY_REMOVE(f0, cast(NULL AS INT))",
+ new Integer[] {1, 2, 2},
+ DataTypes.ARRAY(DataTypes.INT()).nullable())
+ // ARRAY<INT> of null value
+ .testResult(
+ $("f1").arrayRemove(12),
+ "ARRAY_REMOVE(f1, 12)",
+ null,
+ DataTypes.ARRAY(DataTypes.INT()).nullable())
+ .testResult(
+ $("f1").arrayRemove(null),
+ "ARRAY_REMOVE(f1, NULL)",
+ null,
+ DataTypes.ARRAY(DataTypes.INT()).nullable())
+ // ARRAY<STRING> NOT NULL
+ .testResult(
+ $("f2").arrayRemove("Hello"),
+ "ARRAY_REMOVE(f2, 'Hello')",
+ new String[] {"World"},
+ DataTypes.ARRAY(DataTypes.STRING()).notNull())
+ .testResult(
+ $("f2").arrayRemove(
+ lit(null, DataTypes.STRING())
+ .cast(DataTypes.STRING())),
+ "ARRAY_REMOVE(f2, cast(NULL AS VARCHAR))",
Review Comment:
```suggestion
"ARRAY_REMOVE(f2, CAST(NULL AS VARCHAR))",
```
All SQL keywords should be in uppercase
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] liuyongvs commented on pull request #22144: [FLINK-31102][table] Add ARRAY_REMOVE function.
Posted by "liuyongvs (via GitHub)" <gi...@apache.org>.
liuyongvs commented on PR #22144:
URL: https://github.com/apache/flink/pull/22144#issuecomment-1469409145
hi @snuyanzin could you help review it again?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] snuyanzin merged pull request #22144: [FLINK-31102][table] Add ARRAY_REMOVE function.
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin merged PR #22144:
URL: https://github.com/apache/flink/pull/22144
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] liuyongvs commented on a diff in pull request #22144: [FLINK-31102][table] Add ARRAY_REMOVE function.
Posted by "liuyongvs (via GitHub)" <gi...@apache.org>.
liuyongvs commented on code in PR #22144:
URL: https://github.com/apache/flink/pull/22144#discussion_r1135001220
##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/CollectionFunctionsITCase.java:
##########
@@ -178,6 +181,137 @@ Stream<TestSetSpec> getTestSetSpecs() {
null
},
DataTypes.ARRAY(
- DataTypes.ROW(DataTypes.BOOLEAN(), DataTypes.DATE()))));
+ DataTypes.ROW(DataTypes.BOOLEAN(), DataTypes.DATE()))),
+ TestSetSpec.forFunction(BuiltInFunctionDefinitions.ARRAY_REMOVE)
+ .onFieldsWithData(
+ new Integer[] {1, 2, 2},
+ null,
+ new String[] {"Hello", "World"},
+ new Row[] {
+ Row.of(true, LocalDate.of(2022, 4, 20)),
+ Row.of(true, LocalDate.of(1990, 10, 14)),
+ null
+ },
+ new Integer[] {null, null, 1},
+ new Integer[][] {
+ new Integer[] {1, null, 3}, new Integer[] {0}, new Integer[] {1}
+ },
+ new Map[] {
+ CollectionUtil.map(entry(1, "a"), entry(2, "b")),
+ CollectionUtil.map(entry(3, "c"), entry(4, "d")),
+ null
+ })
+ .andDataTypes(
+ DataTypes.ARRAY(DataTypes.INT()),
+ DataTypes.ARRAY(DataTypes.INT()),
+ DataTypes.ARRAY(DataTypes.STRING()).notNull(),
+ DataTypes.ARRAY(
+ DataTypes.ROW(DataTypes.BOOLEAN(), DataTypes.DATE())),
+ DataTypes.ARRAY(DataTypes.INT()),
+ DataTypes.ARRAY(DataTypes.ARRAY(DataTypes.INT())),
+ DataTypes.ARRAY(DataTypes.MAP(DataTypes.INT(), DataTypes.STRING())))
+ // ARRAY<INT>
+ .testResult(
+ $("f0").arrayRemove(2),
+ "ARRAY_REMOVE(f0, 2)",
+ new Integer[] {1},
+ DataTypes.ARRAY(DataTypes.INT()).nullable())
+ .testResult(
+ $("f0").arrayRemove(42),
+ "ARRAY_REMOVE(f0, 42)",
+ new Integer[] {1, 2, 2},
+ DataTypes.ARRAY(DataTypes.INT()).nullable())
+ .testResult(
+ $("f0").arrayRemove(
+ lit(null, DataTypes.SMALLINT())
+ .cast(DataTypes.INT())),
+ "ARRAY_REMOVE(f0, cast(NULL AS INT))",
+ new Integer[] {1, 2, 2},
+ DataTypes.ARRAY(DataTypes.INT()).nullable())
+ // ARRAY<INT> of null value
+ .testResult(
+ $("f1").arrayRemove(12),
+ "ARRAY_REMOVE(f1, 12)",
+ null,
+ DataTypes.ARRAY(DataTypes.INT()).nullable())
+ .testResult(
+ $("f1").arrayRemove(null),
+ "ARRAY_REMOVE(f1, NULL)",
+ null,
+ DataTypes.ARRAY(DataTypes.INT()).nullable())
+ // ARRAY<STRING> NOT NULL
+ .testResult(
+ $("f2").arrayRemove("Hello"),
+ "ARRAY_REMOVE(f2, 'Hello')",
+ new String[] {"World"},
+ DataTypes.ARRAY(DataTypes.STRING()).notNull())
+ .testResult(
+ $("f2").arrayRemove(
+ lit(null, DataTypes.STRING())
+ .cast(DataTypes.STRING())),
+ "ARRAY_REMOVE(f2, cast(NULL AS VARCHAR))",
+ new String[] {"Hello", "World"},
+ DataTypes.ARRAY(DataTypes.STRING()).notNull())
+ // ARRAY<ROW<BOOLEAN, DATE>>
+ .testResult(
+ $("f3").arrayRemove(row(true, LocalDate.of(1990, 10, 14))),
+ "ARRAY_REMOVE(f3, (TRUE, DATE '1990-10-14'))",
+ new Row[] {Row.of(true, LocalDate.of(2022, 4, 20)), null},
+ DataTypes.ARRAY(
+ DataTypes.ROW(
+ DataTypes.BOOLEAN(), DataTypes.DATE()))
+ .nullable())
+ .testResult(
+ $("f3").arrayRemove(row(false, LocalDate.of(1990, 10, 14))),
+ "ARRAY_REMOVE(f3, (FALSE, DATE '1990-10-14'))",
+ new Row[] {
+ Row.of(true, LocalDate.of(2022, 4, 20)),
+ Row.of(true, LocalDate.of(1990, 10, 14)),
+ null
+ },
+ DataTypes.ARRAY(
+ DataTypes.ROW(
+ DataTypes.BOOLEAN(), DataTypes.DATE()))
+ .nullable())
Review Comment:
have done @snuyanzin
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] snuyanzin commented on a diff in pull request #22144: [FLINK-31102][table] Add ARRAY_REMOVE function.
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #22144:
URL: https://github.com/apache/flink/pull/22144#discussion_r1142751304
##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/CollectionFunctionsITCase.java:
##########
@@ -178,6 +181,125 @@ Stream<TestSetSpec> getTestSetSpecs() {
null
},
DataTypes.ARRAY(
- DataTypes.ROW(DataTypes.BOOLEAN(), DataTypes.DATE()))));
+ DataTypes.ROW(DataTypes.BOOLEAN(), DataTypes.DATE()))),
+ TestSetSpec.forFunction(BuiltInFunctionDefinitions.ARRAY_REMOVE)
+ .onFieldsWithData(
+ new Integer[] {1, 2, 2},
+ null,
+ new String[] {"Hello", "World"},
+ new Row[] {
+ Row.of(true, LocalDate.of(2022, 4, 20)),
+ Row.of(true, LocalDate.of(1990, 10, 14)),
+ null
+ },
+ new Integer[] {null, null, 1},
+ new Integer[][] {
+ new Integer[] {1, null, 3}, new Integer[] {0}, new Integer[] {1}
+ },
+ new Map[] {
+ CollectionUtil.map(entry(1, "a"), entry(2, "b")),
+ CollectionUtil.map(entry(3, "c"), entry(4, "d")),
+ null
+ })
+ .andDataTypes(
+ DataTypes.ARRAY(DataTypes.INT()),
+ DataTypes.ARRAY(DataTypes.INT()),
+ DataTypes.ARRAY(DataTypes.STRING()).notNull(),
+ DataTypes.ARRAY(
+ DataTypes.ROW(DataTypes.BOOLEAN(), DataTypes.DATE())),
+ DataTypes.ARRAY(DataTypes.INT()),
+ DataTypes.ARRAY(DataTypes.ARRAY(DataTypes.INT())),
+ DataTypes.ARRAY(DataTypes.MAP(DataTypes.INT(), DataTypes.STRING())))
+ // ARRAY<INT>
+ .testResult(
+ $("f0").arrayRemove(2),
+ "ARRAY_REMOVE(f0, 2)",
+ new Integer[] {1},
+ DataTypes.ARRAY(DataTypes.INT()).nullable())
+ .testResult(
+ $("f0").arrayRemove(42),
+ "ARRAY_REMOVE(f0, 42)",
+ new Integer[] {1, 2, 2},
+ DataTypes.ARRAY(DataTypes.INT()).nullable())
+ .testResult(
+ $("f0").arrayRemove(
+ lit(null, DataTypes.SMALLINT())
+ .cast(DataTypes.INT())),
+ "ARRAY_REMOVE(f0, cast(NULL AS INT))",
+ new Integer[] {1, 2, 2},
+ DataTypes.ARRAY(DataTypes.INT()).nullable())
+ // ARRAY<INT> of null value
+ .testResult(
+ $("f1").arrayRemove(12),
+ "ARRAY_REMOVE(f1, 12)",
+ null,
+ DataTypes.ARRAY(DataTypes.INT()).nullable())
+ .testResult(
+ $("f1").arrayRemove(null),
+ "ARRAY_REMOVE(f1, NULL)",
+ null,
+ DataTypes.ARRAY(DataTypes.INT()).nullable())
+ // ARRAY<STRING> NOT NULL
+ .testResult(
+ $("f2").arrayRemove("Hello"),
+ "ARRAY_REMOVE(f2, 'Hello')",
+ new String[] {"World"},
+ DataTypes.ARRAY(DataTypes.STRING()).notNull())
+ .testResult(
+ $("f2").arrayRemove(
+ lit(null, DataTypes.STRING())
+ .cast(DataTypes.STRING())),
+ "ARRAY_REMOVE(f2, cast(NULL AS VARCHAR))",
+ new String[] {"Hello", "World"},
+ DataTypes.ARRAY(DataTypes.STRING()).notNull())
+ // ARRAY<ROW<BOOLEAN, DATE>>
+ .testResult(
+ $("f3").arrayRemove(row(true, LocalDate.of(1990, 10, 14))),
+ "ARRAY_REMOVE(f3, (TRUE, DATE '1990-10-14'))",
+ new Row[] {Row.of(true, LocalDate.of(2022, 4, 20)), null},
+ DataTypes.ARRAY(
+ DataTypes.ROW(
+ DataTypes.BOOLEAN(), DataTypes.DATE()))
+ .nullable())
+ .testResult(
+ $("f3").arrayRemove(null),
+ "ARRAY_REMOVE(f3, null)",
+ new Row[] {
+ Row.of(true, LocalDate.of(2022, 4, 20)),
+ Row.of(true, LocalDate.of(1990, 10, 14)),
+ },
+ DataTypes.ARRAY(
+ DataTypes.ROW(
+ DataTypes.BOOLEAN(), DataTypes.DATE()))
+ .nullable())
+ // ARRAY<INT> with null elements
Review Comment:
```suggestion
// ARRAY<INT> with NULL elements
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] snuyanzin commented on a diff in pull request #22144: [FLINK-31102][table] Add ARRAY_REMOVE function.
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #22144:
URL: https://github.com/apache/flink/pull/22144#discussion_r1142750735
##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/CollectionFunctionsITCase.java:
##########
@@ -178,6 +181,125 @@ Stream<TestSetSpec> getTestSetSpecs() {
null
},
DataTypes.ARRAY(
- DataTypes.ROW(DataTypes.BOOLEAN(), DataTypes.DATE()))));
+ DataTypes.ROW(DataTypes.BOOLEAN(), DataTypes.DATE()))),
+ TestSetSpec.forFunction(BuiltInFunctionDefinitions.ARRAY_REMOVE)
+ .onFieldsWithData(
+ new Integer[] {1, 2, 2},
+ null,
+ new String[] {"Hello", "World"},
+ new Row[] {
+ Row.of(true, LocalDate.of(2022, 4, 20)),
+ Row.of(true, LocalDate.of(1990, 10, 14)),
+ null
+ },
+ new Integer[] {null, null, 1},
+ new Integer[][] {
+ new Integer[] {1, null, 3}, new Integer[] {0}, new Integer[] {1}
+ },
+ new Map[] {
+ CollectionUtil.map(entry(1, "a"), entry(2, "b")),
+ CollectionUtil.map(entry(3, "c"), entry(4, "d")),
+ null
+ })
+ .andDataTypes(
+ DataTypes.ARRAY(DataTypes.INT()),
+ DataTypes.ARRAY(DataTypes.INT()),
+ DataTypes.ARRAY(DataTypes.STRING()).notNull(),
+ DataTypes.ARRAY(
+ DataTypes.ROW(DataTypes.BOOLEAN(), DataTypes.DATE())),
+ DataTypes.ARRAY(DataTypes.INT()),
+ DataTypes.ARRAY(DataTypes.ARRAY(DataTypes.INT())),
+ DataTypes.ARRAY(DataTypes.MAP(DataTypes.INT(), DataTypes.STRING())))
+ // ARRAY<INT>
+ .testResult(
+ $("f0").arrayRemove(2),
+ "ARRAY_REMOVE(f0, 2)",
+ new Integer[] {1},
+ DataTypes.ARRAY(DataTypes.INT()).nullable())
+ .testResult(
+ $("f0").arrayRemove(42),
+ "ARRAY_REMOVE(f0, 42)",
+ new Integer[] {1, 2, 2},
+ DataTypes.ARRAY(DataTypes.INT()).nullable())
+ .testResult(
+ $("f0").arrayRemove(
+ lit(null, DataTypes.SMALLINT())
+ .cast(DataTypes.INT())),
+ "ARRAY_REMOVE(f0, cast(NULL AS INT))",
+ new Integer[] {1, 2, 2},
+ DataTypes.ARRAY(DataTypes.INT()).nullable())
+ // ARRAY<INT> of null value
+ .testResult(
+ $("f1").arrayRemove(12),
+ "ARRAY_REMOVE(f1, 12)",
+ null,
+ DataTypes.ARRAY(DataTypes.INT()).nullable())
+ .testResult(
+ $("f1").arrayRemove(null),
+ "ARRAY_REMOVE(f1, NULL)",
+ null,
+ DataTypes.ARRAY(DataTypes.INT()).nullable())
+ // ARRAY<STRING> NOT NULL
+ .testResult(
+ $("f2").arrayRemove("Hello"),
+ "ARRAY_REMOVE(f2, 'Hello')",
+ new String[] {"World"},
+ DataTypes.ARRAY(DataTypes.STRING()).notNull())
+ .testResult(
+ $("f2").arrayRemove(
+ lit(null, DataTypes.STRING())
+ .cast(DataTypes.STRING())),
+ "ARRAY_REMOVE(f2, cast(NULL AS VARCHAR))",
+ new String[] {"Hello", "World"},
+ DataTypes.ARRAY(DataTypes.STRING()).notNull())
+ // ARRAY<ROW<BOOLEAN, DATE>>
+ .testResult(
+ $("f3").arrayRemove(row(true, LocalDate.of(1990, 10, 14))),
+ "ARRAY_REMOVE(f3, (TRUE, DATE '1990-10-14'))",
+ new Row[] {Row.of(true, LocalDate.of(2022, 4, 20)), null},
+ DataTypes.ARRAY(
+ DataTypes.ROW(
+ DataTypes.BOOLEAN(), DataTypes.DATE()))
+ .nullable())
+ .testResult(
+ $("f3").arrayRemove(null),
+ "ARRAY_REMOVE(f3, null)",
Review Comment:
```suggestion
"ARRAY_REMOVE(f3, NULL)",
```
##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/CollectionFunctionsITCase.java:
##########
@@ -178,6 +181,125 @@ Stream<TestSetSpec> getTestSetSpecs() {
null
},
DataTypes.ARRAY(
- DataTypes.ROW(DataTypes.BOOLEAN(), DataTypes.DATE()))));
+ DataTypes.ROW(DataTypes.BOOLEAN(), DataTypes.DATE()))),
+ TestSetSpec.forFunction(BuiltInFunctionDefinitions.ARRAY_REMOVE)
+ .onFieldsWithData(
+ new Integer[] {1, 2, 2},
+ null,
+ new String[] {"Hello", "World"},
+ new Row[] {
+ Row.of(true, LocalDate.of(2022, 4, 20)),
+ Row.of(true, LocalDate.of(1990, 10, 14)),
+ null
+ },
+ new Integer[] {null, null, 1},
+ new Integer[][] {
+ new Integer[] {1, null, 3}, new Integer[] {0}, new Integer[] {1}
+ },
+ new Map[] {
+ CollectionUtil.map(entry(1, "a"), entry(2, "b")),
+ CollectionUtil.map(entry(3, "c"), entry(4, "d")),
+ null
+ })
+ .andDataTypes(
+ DataTypes.ARRAY(DataTypes.INT()),
+ DataTypes.ARRAY(DataTypes.INT()),
+ DataTypes.ARRAY(DataTypes.STRING()).notNull(),
+ DataTypes.ARRAY(
+ DataTypes.ROW(DataTypes.BOOLEAN(), DataTypes.DATE())),
+ DataTypes.ARRAY(DataTypes.INT()),
+ DataTypes.ARRAY(DataTypes.ARRAY(DataTypes.INT())),
+ DataTypes.ARRAY(DataTypes.MAP(DataTypes.INT(), DataTypes.STRING())))
+ // ARRAY<INT>
+ .testResult(
+ $("f0").arrayRemove(2),
+ "ARRAY_REMOVE(f0, 2)",
+ new Integer[] {1},
+ DataTypes.ARRAY(DataTypes.INT()).nullable())
+ .testResult(
+ $("f0").arrayRemove(42),
+ "ARRAY_REMOVE(f0, 42)",
+ new Integer[] {1, 2, 2},
+ DataTypes.ARRAY(DataTypes.INT()).nullable())
+ .testResult(
+ $("f0").arrayRemove(
+ lit(null, DataTypes.SMALLINT())
+ .cast(DataTypes.INT())),
+ "ARRAY_REMOVE(f0, cast(NULL AS INT))",
+ new Integer[] {1, 2, 2},
+ DataTypes.ARRAY(DataTypes.INT()).nullable())
+ // ARRAY<INT> of null value
+ .testResult(
+ $("f1").arrayRemove(12),
+ "ARRAY_REMOVE(f1, 12)",
+ null,
+ DataTypes.ARRAY(DataTypes.INT()).nullable())
+ .testResult(
+ $("f1").arrayRemove(null),
+ "ARRAY_REMOVE(f1, NULL)",
+ null,
+ DataTypes.ARRAY(DataTypes.INT()).nullable())
+ // ARRAY<STRING> NOT NULL
+ .testResult(
+ $("f2").arrayRemove("Hello"),
+ "ARRAY_REMOVE(f2, 'Hello')",
+ new String[] {"World"},
+ DataTypes.ARRAY(DataTypes.STRING()).notNull())
+ .testResult(
+ $("f2").arrayRemove(
+ lit(null, DataTypes.STRING())
+ .cast(DataTypes.STRING())),
+ "ARRAY_REMOVE(f2, cast(NULL AS VARCHAR))",
+ new String[] {"Hello", "World"},
+ DataTypes.ARRAY(DataTypes.STRING()).notNull())
+ // ARRAY<ROW<BOOLEAN, DATE>>
+ .testResult(
+ $("f3").arrayRemove(row(true, LocalDate.of(1990, 10, 14))),
+ "ARRAY_REMOVE(f3, (TRUE, DATE '1990-10-14'))",
+ new Row[] {Row.of(true, LocalDate.of(2022, 4, 20)), null},
+ DataTypes.ARRAY(
+ DataTypes.ROW(
+ DataTypes.BOOLEAN(), DataTypes.DATE()))
+ .nullable())
+ .testResult(
+ $("f3").arrayRemove(null),
+ "ARRAY_REMOVE(f3, null)",
+ new Row[] {
+ Row.of(true, LocalDate.of(2022, 4, 20)),
+ Row.of(true, LocalDate.of(1990, 10, 14)),
+ },
+ DataTypes.ARRAY(
+ DataTypes.ROW(
+ DataTypes.BOOLEAN(), DataTypes.DATE()))
+ .nullable())
+ // ARRAY<INT> with null elements
+ .testResult(
+ $("f4").arrayRemove(null),
+ "ARRAY_REMOVE(f4, NULL)",
+ new Integer[] {1},
+ DataTypes.ARRAY(DataTypes.INT()).nullable())
+ // ARRAY<ARRAY<INT>>
+ .testResult(
+ $("f5").arrayRemove(new Integer[] {0}),
+ "ARRAY_REMOVE(f5, array[0])",
Review Comment:
```suggestion
"ARRAY_REMOVE(f5, ARRAY[0])",
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] snuyanzin commented on a diff in pull request #22144: [FLINK-31102][table] Add ARRAY_REMOVE function.
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #22144:
URL: https://github.com/apache/flink/pull/22144#discussion_r1139401795
##########
docs/data/sql_functions.yml:
##########
@@ -617,6 +617,9 @@ collection:
- sql: ARRAY_DISTINCT(haystack)
table: haystack.arrayDistinct()
description: Returns an array with unique elements. If the array itself is null, the function will return null. Keeps ordering of elements.
+ - sql: ARRAY_REMOVE(haystack, needle)
+ table: haystack.arrayRemove(needle)
+ description: Remove all elements that equal to element from array. If the array itself is null, the function will return null. Keeps ordering of elements.
Review Comment:
Sorry, looks like missed it
```suggestion
description: Removes all elements that equal to element from array. If the array itself is null, the function will return null. Keeps ordering of elements.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] liuyongvs commented on a diff in pull request #22144: [FLINK-31102][table] Add ARRAY_REMOVE function.
Posted by "liuyongvs (via GitHub)" <gi...@apache.org>.
liuyongvs commented on code in PR #22144:
URL: https://github.com/apache/flink/pull/22144#discussion_r1139573794
##########
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/internal/BaseExpressions.java:
##########
@@ -1359,6 +1360,16 @@ public OutType arrayDistinct() {
return toApiSpecificExpression(unresolvedCall(ARRAY_DISTINCT, toExpr()));
}
+ /**
+ * Remove all elements that equal to element from array.
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] liuyongvs commented on pull request #22144: [FLINK-31102][table] Add ARRAY_REMOVE function.
Posted by "liuyongvs (via GitHub)" <gi...@apache.org>.
liuyongvs commented on PR #22144:
URL: https://github.com/apache/flink/pull/22144#issuecomment-1473012377
@snuyanzin rebase to fix the conflicts and reviews from you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org