You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "twalthr (via GitHub)" <gi...@apache.org> on 2025/01/08 13:58:07 UTC
[PR] [FLINK-36707][table] Integrate PTF type inference into Calcite [flink]
twalthr opened a new pull request, #25928:
URL: https://github.com/apache/flink/pull/25928
## What is the purpose of the change
This integrates the new type inference of PTFs into Calcite and the existing planner classes around operand checking and return type inference. In particular the new inference is able to deal with table arguments coming from `StaticArgument` signatures. Some historical shortcomings have been fixed on the way.
## Brief change log
- Introduce a system type inference within the planner for all functions
- Let the system type inference wrap atomic types into rows if necessary for (process) table functions
- Support table arguments for both tables with row and set semantics.
- Add extensive validation for incorrect parameters
## Verifying this change
This change added tests and can be verified as follows: `ProcessTableFunctionTest`
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? yes
- If yes, how is the feature documented? JavaDocs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-36707][table] Integrate PTF type inference into Calcite [flink]
Posted by "dawidwys (via GitHub)" <gi...@apache.org>.
dawidwys commented on code in PR #25928:
URL: https://github.com/apache/flink/pull/25928#discussion_r1907380281
##########
flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/inference/SystemTypeInference.java:
##########
@@ -0,0 +1,324 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.types.inference;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.api.DataTypes.Field;
+import org.apache.flink.table.api.ValidationException;
+import org.apache.flink.table.functions.FunctionDefinition;
+import org.apache.flink.table.functions.FunctionKind;
+import org.apache.flink.table.functions.ProcessTableFunction;
+import org.apache.flink.table.functions.TableSemantics;
+import org.apache.flink.table.types.DataType;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.Set;
+import java.util.function.Predicate;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+/**
+ * Extends the {@link TypeInference} function-aware by additional system columns and validation.
+ *
+ * <p>During planning system columns are available and can be accessed in SQL, during runtime those
+ * columns are not passed or returned by the eval() method. They are handled with custom code paths.
+ *
+ * <p>For example, for {@link ProcessTableFunction}, this utility class implicitly adds the optional
+ * {@code uid} and {@code on_time} args and an additional {@code rowtime} column in the output.
+ * Additionally, it adds a validation layer for complex {@link StaticArgument}s.
+ */
+@Internal
+public class SystemTypeInference {
+
+ private static final List<StaticArgument> PROCESS_TABLE_FUNCTION_SYSTEM_ARGS =
+ List.of(StaticArgument.scalar("uid", DataTypes.STRING(), true));
+
+ /** Format of unique identifiers for {@link ProcessTableFunction}. */
+ private static final Predicate<String> UID_FORMAT =
+ Pattern.compile("^[a-zA-Z_][a-zA-Z-_0-9]*$").asPredicate();
+
+ public static TypeInference of(FunctionKind functionKind, TypeInference origin) {
+ final TypeInference.Builder builder = TypeInference.newBuilder();
+
+ final List<StaticArgument> defaultArgs =
+ applyDefaultArgs(builder, functionKind, origin.getStaticArguments().orElse(null));
+ builder.inputTypeStrategy(origin.getInputTypeStrategy());
+ builder.stateTypeStrategies(origin.getStateTypeStrategies());
+ builder.outputTypeStrategy(origin.getOutputTypeStrategy());
+
+ final List<StaticArgument> systemArgs = applySystemArgs(builder, functionKind, defaultArgs);
+ applySystemInputStrategy(builder, functionKind, systemArgs, origin);
+ applySystemOutputStrategy(builder, functionKind, origin);
+
+ return builder.build();
+ }
+
+ // --------------------------------------------------------------------------------------------
+
+ private static @Nullable List<StaticArgument> applyDefaultArgs(
Review Comment:
nit: is the return type necessary? We nver modify the input `defaultArgs`.
##########
flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/inference/SystemTypeInference.java:
##########
@@ -0,0 +1,324 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.types.inference;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.api.DataTypes.Field;
+import org.apache.flink.table.api.ValidationException;
+import org.apache.flink.table.functions.FunctionDefinition;
+import org.apache.flink.table.functions.FunctionKind;
+import org.apache.flink.table.functions.ProcessTableFunction;
+import org.apache.flink.table.functions.TableSemantics;
+import org.apache.flink.table.types.DataType;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.Set;
+import java.util.function.Predicate;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+/**
+ * Extends the {@link TypeInference} function-aware by additional system columns and validation.
+ *
+ * <p>During planning system columns are available and can be accessed in SQL, during runtime those
+ * columns are not passed or returned by the eval() method. They are handled with custom code paths.
+ *
+ * <p>For example, for {@link ProcessTableFunction}, this utility class implicitly adds the optional
+ * {@code uid} and {@code on_time} args and an additional {@code rowtime} column in the output.
+ * Additionally, it adds a validation layer for complex {@link StaticArgument}s.
+ */
+@Internal
+public class SystemTypeInference {
+
+ private static final List<StaticArgument> PROCESS_TABLE_FUNCTION_SYSTEM_ARGS =
+ List.of(StaticArgument.scalar("uid", DataTypes.STRING(), true));
+
+ /** Format of unique identifiers for {@link ProcessTableFunction}. */
+ private static final Predicate<String> UID_FORMAT =
+ Pattern.compile("^[a-zA-Z_][a-zA-Z-_0-9]*$").asPredicate();
+
+ public static TypeInference of(FunctionKind functionKind, TypeInference origin) {
+ final TypeInference.Builder builder = TypeInference.newBuilder();
+
+ final List<StaticArgument> defaultArgs =
+ applyDefaultArgs(builder, functionKind, origin.getStaticArguments().orElse(null));
+ builder.inputTypeStrategy(origin.getInputTypeStrategy());
+ builder.stateTypeStrategies(origin.getStateTypeStrategies());
+ builder.outputTypeStrategy(origin.getOutputTypeStrategy());
+
+ final List<StaticArgument> systemArgs = applySystemArgs(builder, functionKind, defaultArgs);
+ applySystemInputStrategy(builder, functionKind, systemArgs, origin);
+ applySystemOutputStrategy(builder, functionKind, origin);
+
+ return builder.build();
+ }
+
+ // --------------------------------------------------------------------------------------------
+
+ private static @Nullable List<StaticArgument> applyDefaultArgs(
+ TypeInference.Builder builder,
+ FunctionKind functionKind,
+ @Nullable List<StaticArgument> defaultArgs) {
+ if (defaultArgs == null) {
+ return null;
+ }
+ if (functionKind != FunctionKind.PROCESS_TABLE) {
+ checkScalarArgsOnly(defaultArgs);
+ }
+ builder.staticArguments(defaultArgs);
Review Comment:
nit: I believe you could set it only once in `applySystemArgs`
##########
flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/inference/SystemTypeInference.java:
##########
@@ -0,0 +1,324 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.types.inference;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.api.DataTypes.Field;
+import org.apache.flink.table.api.ValidationException;
+import org.apache.flink.table.functions.FunctionDefinition;
+import org.apache.flink.table.functions.FunctionKind;
+import org.apache.flink.table.functions.ProcessTableFunction;
+import org.apache.flink.table.functions.TableSemantics;
+import org.apache.flink.table.types.DataType;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.Set;
+import java.util.function.Predicate;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+/**
+ * Extends the {@link TypeInference} function-aware by additional system columns and validation.
+ *
+ * <p>During planning system columns are available and can be accessed in SQL, during runtime those
+ * columns are not passed or returned by the eval() method. They are handled with custom code paths.
+ *
+ * <p>For example, for {@link ProcessTableFunction}, this utility class implicitly adds the optional
+ * {@code uid} and {@code on_time} args and an additional {@code rowtime} column in the output.
+ * Additionally, it adds a validation layer for complex {@link StaticArgument}s.
+ */
+@Internal
+public class SystemTypeInference {
+
+ private static final List<StaticArgument> PROCESS_TABLE_FUNCTION_SYSTEM_ARGS =
+ List.of(StaticArgument.scalar("uid", DataTypes.STRING(), true));
+
+ /** Format of unique identifiers for {@link ProcessTableFunction}. */
+ private static final Predicate<String> UID_FORMAT =
+ Pattern.compile("^[a-zA-Z_][a-zA-Z-_0-9]*$").asPredicate();
+
+ public static TypeInference of(FunctionKind functionKind, TypeInference origin) {
+ final TypeInference.Builder builder = TypeInference.newBuilder();
+
+ final List<StaticArgument> defaultArgs =
+ applyDefaultArgs(builder, functionKind, origin.getStaticArguments().orElse(null));
+ builder.inputTypeStrategy(origin.getInputTypeStrategy());
+ builder.stateTypeStrategies(origin.getStateTypeStrategies());
+ builder.outputTypeStrategy(origin.getOutputTypeStrategy());
+
+ final List<StaticArgument> systemArgs = applySystemArgs(builder, functionKind, defaultArgs);
+ applySystemInputStrategy(builder, functionKind, systemArgs, origin);
+ applySystemOutputStrategy(builder, functionKind, origin);
+
+ return builder.build();
+ }
+
+ // --------------------------------------------------------------------------------------------
+
+ private static @Nullable List<StaticArgument> applyDefaultArgs(
+ TypeInference.Builder builder,
+ FunctionKind functionKind,
+ @Nullable List<StaticArgument> defaultArgs) {
+ if (defaultArgs == null) {
+ return null;
+ }
+ if (functionKind != FunctionKind.PROCESS_TABLE) {
+ checkScalarArgsOnly(defaultArgs);
+ }
+ builder.staticArguments(defaultArgs);
+ return defaultArgs;
+ }
+
+ private static void checkScalarArgsOnly(List<StaticArgument> defaultArgs) {
+ defaultArgs.forEach(
+ arg -> {
+ if (!arg.is(StaticArgumentTrait.SCALAR)) {
+ throw new ValidationException(
+ String.format(
+ "Only scalar arguments are supported at this location. "
+ + "But argument '%s' declared the following traits: %s",
+ arg.getName(), arg.getTraits()));
+ }
+ });
+ }
+
+ private static @Nullable List<StaticArgument> applySystemArgs(
+ TypeInference.Builder builder,
+ FunctionKind functionKind,
+ @Nullable List<StaticArgument> defaultArgs) {
+ if (functionKind != FunctionKind.PROCESS_TABLE) {
+ return defaultArgs;
+ }
+ if (defaultArgs == null) {
+ throw new ValidationException(
+ "Function requires a static signature that is not overloaded and doesn't contain varargs.");
+ }
+
+ checkReservedArgs(defaultArgs);
+
+ final List<StaticArgument> newStaticArgs = new ArrayList<>(defaultArgs);
+ newStaticArgs.addAll(PROCESS_TABLE_FUNCTION_SYSTEM_ARGS);
+ builder.staticArguments(newStaticArgs);
+ return newStaticArgs;
+ }
+
+ private static void checkReservedArgs(List<StaticArgument> staticArgs) {
+ final Set<String> declaredArgs =
+ staticArgs.stream().map(StaticArgument::getName).collect(Collectors.toSet());
+ final Set<String> reservedArgs =
+ PROCESS_TABLE_FUNCTION_SYSTEM_ARGS.stream()
+ .map(StaticArgument::getName)
+ .collect(Collectors.toSet());
+ if (reservedArgs.stream().anyMatch(declaredArgs::contains)) {
+ throw new ValidationException(
+ "Function signature must not declare system arguments. "
+ + "Reserved argument names are: "
+ + reservedArgs);
+ }
+ }
+
+ private static void applySystemInputStrategy(
+ TypeInference.Builder builder,
+ FunctionKind functionKind,
+ @Nullable List<StaticArgument> staticArgs,
+ TypeInference origin) {
+ if (functionKind != FunctionKind.PROCESS_TABLE) {
+ return;
+ }
+ builder.inputTypeStrategy(
+ new SystemInputStrategy(staticArgs, origin.getInputTypeStrategy()));
+ }
+
+ private static void applySystemOutputStrategy(
+ TypeInference.Builder builder, FunctionKind functionKind, TypeInference origin) {
+ if (functionKind != FunctionKind.TABLE && functionKind != FunctionKind.PROCESS_TABLE) {
+ return;
+ }
+ builder.outputTypeStrategy(new SystemOutputStrategy(origin.getOutputTypeStrategy()));
+ }
+
+ private static class SystemOutputStrategy implements TypeStrategy {
+
+ private final TypeStrategy origin;
+
+ private SystemOutputStrategy(TypeStrategy origin) {
+ this.origin = origin;
+ }
+
+ @Override
+ public Optional<DataType> inferType(CallContext callContext) {
+ return origin.inferType(callContext)
+ .map(
+ dataType -> {
+ final List<DataType> fieldTypes =
+ DataType.getFieldDataTypes(dataType);
+ final List<String> fieldNames = DataType.getFieldNames(dataType);
+ final List<Field> fields = new ArrayList<>();
+ if (fieldTypes.isEmpty()) {
+ // Before the system type inference was introduced, SQL and
+ // Table API chose a different default field name.
+ // EXPR$0 is chosen for best-effort backwards compatibility for
+ // SQL users.
+ fields.add(DataTypes.FIELD("EXPR$0", dataType));
+ } else {
+ IntStream.range(0, fieldTypes.size())
+ .mapToObj(
+ pos ->
+ DataTypes.FIELD(
+ fieldNames.get(pos),
+ fieldTypes.get(pos)))
+ .forEach(fields::add);
+ }
+
+ return DataTypes.ROW(fields).notNull();
+ });
+ }
+ }
+
+ private static class SystemInputStrategy implements InputTypeStrategy {
+
+ private final List<StaticArgument> staticArgs;
+ private final InputTypeStrategy origin;
+
+ private SystemInputStrategy(List<StaticArgument> staticArgs, InputTypeStrategy origin) {
+ this.staticArgs = staticArgs;
+ this.origin = origin;
+ }
+
+ @Override
+ public ArgumentCount getArgumentCount() {
+ // Static arguments declare the count
+ return InputTypeStrategies.WILDCARD.getArgumentCount();
Review Comment:
Shouldn't you return the count of static arguments then?
##########
flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/inference/SystemTypeInference.java:
##########
@@ -0,0 +1,324 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.types.inference;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.api.DataTypes.Field;
+import org.apache.flink.table.api.ValidationException;
+import org.apache.flink.table.functions.FunctionDefinition;
+import org.apache.flink.table.functions.FunctionKind;
+import org.apache.flink.table.functions.ProcessTableFunction;
+import org.apache.flink.table.functions.TableSemantics;
+import org.apache.flink.table.types.DataType;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.Set;
+import java.util.function.Predicate;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+/**
+ * Extends the {@link TypeInference} function-aware by additional system columns and validation.
+ *
+ * <p>During planning system columns are available and can be accessed in SQL, during runtime those
+ * columns are not passed or returned by the eval() method. They are handled with custom code paths.
+ *
+ * <p>For example, for {@link ProcessTableFunction}, this utility class implicitly adds the optional
+ * {@code uid} and {@code on_time} args and an additional {@code rowtime} column in the output.
+ * Additionally, it adds a validation layer for complex {@link StaticArgument}s.
+ */
+@Internal
+public class SystemTypeInference {
+
+ private static final List<StaticArgument> PROCESS_TABLE_FUNCTION_SYSTEM_ARGS =
+ List.of(StaticArgument.scalar("uid", DataTypes.STRING(), true));
+
+ /** Format of unique identifiers for {@link ProcessTableFunction}. */
+ private static final Predicate<String> UID_FORMAT =
+ Pattern.compile("^[a-zA-Z_][a-zA-Z-_0-9]*$").asPredicate();
+
+ public static TypeInference of(FunctionKind functionKind, TypeInference origin) {
+ final TypeInference.Builder builder = TypeInference.newBuilder();
+
+ final List<StaticArgument> defaultArgs =
+ applyDefaultArgs(builder, functionKind, origin.getStaticArguments().orElse(null));
+ builder.inputTypeStrategy(origin.getInputTypeStrategy());
+ builder.stateTypeStrategies(origin.getStateTypeStrategies());
+ builder.outputTypeStrategy(origin.getOutputTypeStrategy());
+
+ final List<StaticArgument> systemArgs = applySystemArgs(builder, functionKind, defaultArgs);
+ applySystemInputStrategy(builder, functionKind, systemArgs, origin);
+ applySystemOutputStrategy(builder, functionKind, origin);
+
+ return builder.build();
+ }
+
+ // --------------------------------------------------------------------------------------------
+
+ private static @Nullable List<StaticArgument> applyDefaultArgs(
+ TypeInference.Builder builder,
+ FunctionKind functionKind,
+ @Nullable List<StaticArgument> defaultArgs) {
+ if (defaultArgs == null) {
+ return null;
+ }
+ if (functionKind != FunctionKind.PROCESS_TABLE) {
+ checkScalarArgsOnly(defaultArgs);
+ }
+ builder.staticArguments(defaultArgs);
Review Comment:
nit: Probably a matter of taste, but you could do that for all three: `inputStrategy`, `outputStrategy`, `staticArguments`. First calculate them:
```
if (functionKind == FunctionKind.PROCESS_TABLE) {
return new SystemInput/OutputTypeStrategy(...);
} else {
return origin.getInput/OutputTypeStrategy()
}
```
and then set it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-36707][table] Integrate PTF type inference into Calcite [flink]
Posted by "twalthr (via GitHub)" <gi...@apache.org>.
twalthr commented on code in PR #25928:
URL: https://github.com/apache/flink/pull/25928#discussion_r1908556932
##########
flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/inference/SystemTypeInference.java:
##########
@@ -0,0 +1,324 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.types.inference;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.api.DataTypes.Field;
+import org.apache.flink.table.api.ValidationException;
+import org.apache.flink.table.functions.FunctionDefinition;
+import org.apache.flink.table.functions.FunctionKind;
+import org.apache.flink.table.functions.ProcessTableFunction;
+import org.apache.flink.table.functions.TableSemantics;
+import org.apache.flink.table.types.DataType;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.Set;
+import java.util.function.Predicate;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+/**
+ * Extends the {@link TypeInference} function-aware by additional system columns and validation.
+ *
+ * <p>During planning system columns are available and can be accessed in SQL, during runtime those
+ * columns are not passed or returned by the eval() method. They are handled with custom code paths.
+ *
+ * <p>For example, for {@link ProcessTableFunction}, this utility class implicitly adds the optional
+ * {@code uid} and {@code on_time} args and an additional {@code rowtime} column in the output.
+ * Additionally, it adds a validation layer for complex {@link StaticArgument}s.
+ */
+@Internal
+public class SystemTypeInference {
+
+ private static final List<StaticArgument> PROCESS_TABLE_FUNCTION_SYSTEM_ARGS =
+ List.of(StaticArgument.scalar("uid", DataTypes.STRING(), true));
+
+ /** Format of unique identifiers for {@link ProcessTableFunction}. */
+ private static final Predicate<String> UID_FORMAT =
+ Pattern.compile("^[a-zA-Z_][a-zA-Z-_0-9]*$").asPredicate();
+
+ public static TypeInference of(FunctionKind functionKind, TypeInference origin) {
+ final TypeInference.Builder builder = TypeInference.newBuilder();
+
+ final List<StaticArgument> defaultArgs =
+ applyDefaultArgs(builder, functionKind, origin.getStaticArguments().orElse(null));
+ builder.inputTypeStrategy(origin.getInputTypeStrategy());
+ builder.stateTypeStrategies(origin.getStateTypeStrategies());
+ builder.outputTypeStrategy(origin.getOutputTypeStrategy());
+
+ final List<StaticArgument> systemArgs = applySystemArgs(builder, functionKind, defaultArgs);
+ applySystemInputStrategy(builder, functionKind, systemArgs, origin);
+ applySystemOutputStrategy(builder, functionKind, origin);
+
+ return builder.build();
+ }
+
+ // --------------------------------------------------------------------------------------------
+
+ private static @Nullable List<StaticArgument> applyDefaultArgs(
+ TypeInference.Builder builder,
+ FunctionKind functionKind,
+ @Nullable List<StaticArgument> defaultArgs) {
+ if (defaultArgs == null) {
+ return null;
+ }
+ if (functionKind != FunctionKind.PROCESS_TABLE) {
+ checkScalarArgsOnly(defaultArgs);
+ }
+ builder.staticArguments(defaultArgs);
+ return defaultArgs;
+ }
+
+ private static void checkScalarArgsOnly(List<StaticArgument> defaultArgs) {
+ defaultArgs.forEach(
+ arg -> {
+ if (!arg.is(StaticArgumentTrait.SCALAR)) {
+ throw new ValidationException(
+ String.format(
+ "Only scalar arguments are supported at this location. "
+ + "But argument '%s' declared the following traits: %s",
+ arg.getName(), arg.getTraits()));
+ }
+ });
+ }
+
+ private static @Nullable List<StaticArgument> applySystemArgs(
+ TypeInference.Builder builder,
+ FunctionKind functionKind,
+ @Nullable List<StaticArgument> defaultArgs) {
+ if (functionKind != FunctionKind.PROCESS_TABLE) {
+ return defaultArgs;
+ }
+ if (defaultArgs == null) {
+ throw new ValidationException(
+ "Function requires a static signature that is not overloaded and doesn't contain varargs.");
+ }
+
+ checkReservedArgs(defaultArgs);
+
+ final List<StaticArgument> newStaticArgs = new ArrayList<>(defaultArgs);
+ newStaticArgs.addAll(PROCESS_TABLE_FUNCTION_SYSTEM_ARGS);
+ builder.staticArguments(newStaticArgs);
+ return newStaticArgs;
+ }
+
+ private static void checkReservedArgs(List<StaticArgument> staticArgs) {
+ final Set<String> declaredArgs =
+ staticArgs.stream().map(StaticArgument::getName).collect(Collectors.toSet());
+ final Set<String> reservedArgs =
+ PROCESS_TABLE_FUNCTION_SYSTEM_ARGS.stream()
+ .map(StaticArgument::getName)
+ .collect(Collectors.toSet());
+ if (reservedArgs.stream().anyMatch(declaredArgs::contains)) {
+ throw new ValidationException(
+ "Function signature must not declare system arguments. "
+ + "Reserved argument names are: "
+ + reservedArgs);
+ }
+ }
+
+ private static void applySystemInputStrategy(
+ TypeInference.Builder builder,
+ FunctionKind functionKind,
+ @Nullable List<StaticArgument> staticArgs,
+ TypeInference origin) {
+ if (functionKind != FunctionKind.PROCESS_TABLE) {
+ return;
+ }
+ builder.inputTypeStrategy(
+ new SystemInputStrategy(staticArgs, origin.getInputTypeStrategy()));
+ }
+
+ private static void applySystemOutputStrategy(
+ TypeInference.Builder builder, FunctionKind functionKind, TypeInference origin) {
+ if (functionKind != FunctionKind.TABLE && functionKind != FunctionKind.PROCESS_TABLE) {
+ return;
+ }
+ builder.outputTypeStrategy(new SystemOutputStrategy(origin.getOutputTypeStrategy()));
+ }
+
+ private static class SystemOutputStrategy implements TypeStrategy {
+
+ private final TypeStrategy origin;
+
+ private SystemOutputStrategy(TypeStrategy origin) {
+ this.origin = origin;
+ }
+
+ @Override
+ public Optional<DataType> inferType(CallContext callContext) {
+ return origin.inferType(callContext)
+ .map(
+ dataType -> {
+ final List<DataType> fieldTypes =
+ DataType.getFieldDataTypes(dataType);
+ final List<String> fieldNames = DataType.getFieldNames(dataType);
+ final List<Field> fields = new ArrayList<>();
+ if (fieldTypes.isEmpty()) {
+ // Before the system type inference was introduced, SQL and
+ // Table API chose a different default field name.
+ // EXPR$0 is chosen for best-effort backwards compatibility for
+ // SQL users.
+ fields.add(DataTypes.FIELD("EXPR$0", dataType));
+ } else {
+ IntStream.range(0, fieldTypes.size())
+ .mapToObj(
+ pos ->
+ DataTypes.FIELD(
+ fieldNames.get(pos),
+ fieldTypes.get(pos)))
+ .forEach(fields::add);
+ }
+
+ return DataTypes.ROW(fields).notNull();
+ });
+ }
+ }
+
+ private static class SystemInputStrategy implements InputTypeStrategy {
+
+ private final List<StaticArgument> staticArgs;
+ private final InputTypeStrategy origin;
+
+ private SystemInputStrategy(List<StaticArgument> staticArgs, InputTypeStrategy origin) {
+ this.staticArgs = staticArgs;
+ this.origin = origin;
+ }
+
+ @Override
+ public ArgumentCount getArgumentCount() {
+ // Static arguments declare the count
+ return InputTypeStrategies.WILDCARD.getArgumentCount();
Review Comment:
No this is not important. Static arguments take precedence. The input strategy only serves as a validation layer. Since the count is already validated we don't need to validate it a second time. I will add a comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-36707][table] Integrate PTF type inference into Calcite [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #25928:
URL: https://github.com/apache/flink/pull/25928#discussion_r1907250440
##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/stream/sql/ProcessTableFunctionTest.java:
##########
@@ -0,0 +1,326 @@
+package org.apache.flink.table.planner.plan.stream.sql;
+
+import org.apache.flink.table.annotation.ArgumentHint;
+import org.apache.flink.table.annotation.ArgumentTrait;
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.api.TableConfig;
+import org.apache.flink.table.catalog.DataTypeFactory;
+import org.apache.flink.table.functions.ProcessTableFunction;
+import org.apache.flink.table.functions.ScalarFunction;
+import org.apache.flink.table.functions.TableFunction;
+import org.apache.flink.table.planner.utils.TableTestBase;
+import org.apache.flink.table.planner.utils.TableTestUtil;
+import org.apache.flink.table.types.inference.StaticArgument;
+import org.apache.flink.table.types.inference.StaticArgumentTrait;
+import org.apache.flink.table.types.inference.TypeInference;
+import org.apache.flink.types.Row;
+
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.util.EnumSet;
+import java.util.Optional;
+
+import static org.apache.flink.table.annotation.ArgumentTrait.OPTIONAL_PARTITION_BY;
+import static org.apache.flink.table.annotation.ArgumentTrait.TABLE_AS_ROW;
+import static org.apache.flink.table.annotation.ArgumentTrait.TABLE_AS_SET;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+
+/** Tests for the type inference and planning part of {@link ProcessTableFunction}. */
+public class ProcessTableFunctionTest extends TableTestBase {
+
+ private TableTestUtil util;
+
+ @BeforeEach
+ void setup() {
+ util = streamTestUtil(TableConfig.getDefault());
+ util.tableEnv().executeSql("CREATE VIEW t1 AS SELECT 'Bob' AS name, 12 AS score");
+ util.tableEnv().executeSql("CREATE VIEW t2 AS SELECT 'Bob' AS name, 12 AS different");
+ util.tableEnv().executeSql("CREATE VIEW t3 AS SELECT 'Bob' AS name, TRUE AS isValid");
+ }
+
+ @Test
+ void testScalarArgsNoUid() {
+ util.addTemporarySystemFunction("f", ScalarArgsFunction.class);
+ util.verifyRelPlan("SELECT * FROM f(i => 1, b => true)");
+ }
+
+ @Test
+ void testScalarArgsWithUid() {
+ util.addTemporarySystemFunction("f", ScalarArgsFunction.class);
+ // argument 'uid' is also reordered
+ util.verifyRelPlan("SELECT * FROM f(uid => 'my-uid', i => 1, b => true)");
+ }
+
+ @Test
+ void testUnknownScalarArg() {
+ util.addTemporarySystemFunction("f", ScalarArgsFunction.class);
+ // argument 'invalid' is ignored
+ util.verifyRelPlan("SELECT * FROM f(i => 1, b => true, invalid => 'invalid')");
+ }
+
+ @Test
+ void testInvalidUid() {
+ util.addTemporarySystemFunction("f", ScalarArgsFunction.class);
+ assertThatThrownBy(
+ () -> util.verifyRelPlan("SELECT * FROM f(uid => '%', i => 1, b => true)"))
+ .hasRootCauseMessage(
+ "Invalid unique identifier for process table function. "
+ + "The 'uid' argument must be a string literal that follows the pattern [a-zA-Z_][a-zA-Z-_0-9]*. "
+ + "But found: %");
+ }
+
+ @Test
+ void testTableAsRow() {
+ util.addTemporarySystemFunction("f", TableAsRowFunction.class);
+ assertReachesOptimizer("SELECT * FROM f(r => TABLE t1, i => 1)");
+ }
+
+ @Test
+ void testTypedTableAsRow() {
+ util.addTemporarySystemFunction("f", TypedTableAsRowFunction.class);
+ assertReachesOptimizer("SELECT * FROM f(u => TABLE t1, i => 1)");
+ }
+
+ @Test
+ void testTypedTableAsRowIgnoringColumnNames() {
+ util.addTemporarySystemFunction("f", TypedTableAsRowFunction.class);
+ // function expects <STRING name, INT score>
+ // but table is <STRING name, INT different>
+ assertReachesOptimizer("SELECT * FROM f(u => TABLE t2, i => 1)");
+ }
+
+ @Test
+ void testTypedTableAsRowWithInvalidInput() {
+ util.addTemporarySystemFunction("f", TypedTableAsRowFunction.class);
+ // function expects <STRING name, INT score>
+ assertThatThrownBy(() -> util.verifyRelPlan("SELECT * FROM f(u => TABLE t3, i => 1)"))
+ .hasMessageContaining(
+ "No match found for function signature "
+ + "f(<RecordType(CHAR(3) name, BOOLEAN isValid)>, <NUMERIC>, <CHARACTER>)");
+ }
+
+ @Test
+ void testTableAsSet() {
+ util.addTemporarySystemFunction("f", TableAsSetFunction.class);
+ assertReachesOptimizer("SELECT * FROM f(r => TABLE t1 PARTITION BY name, i => 1)");
+ }
+
+ @Test
+ void testTableAsSetWithInvalidPartitionBy() {
+ util.addTemporarySystemFunction("f", TableAsSetFunction.class);
+ assertThatThrownBy(() -> util.verifyRelPlan("SELECT * FROM f(r => TABLE t1, i => 1)"))
+ .hasRootCauseMessage(
+ "Table argument 'r' requires a PARTITION BY clause for parallel processing.");
+ }
+
+ @Test
+ void testTableAsSetOptionalPartitionBy() {
+ util.addTemporarySystemFunction("f", TableAsSetOptionalPartitionFunction.class);
+ assertReachesOptimizer("SELECT * FROM f(r => TABLE t1, i => 1)");
+ }
+
+ @Test
+ void testTypedTableAsSet() {
+ util.addTemporarySystemFunction("f", TypedTableAsSetFunction.class);
+ assertReachesOptimizer("SELECT * FROM f(u => TABLE t1 PARTITION BY name, i => 1)");
+ }
+
+ @Test
+ void testTypedTableAsSetWithInvalidInput() {
+ util.addTemporarySystemFunction("f", TypedTableAsSetFunction.class);
+ // function expects <STRING name, INT score>
+ assertThatThrownBy(
+ () ->
+ util.verifyRelPlan(
+ "SELECT * FROM f(u => TABLE t3 PARTITION BY name, i => 1)"))
+ .hasMessageContaining(
+ "No match found for function signature "
+ + "f(<RecordType(CHAR(3) name, BOOLEAN isValid)>, <NUMERIC>, <CHARACTER>)");
+ }
+
+ @Test
+ void testEmptyArgs() {
+ util.addTemporarySystemFunction("f", EmptyArgFunction.class);
+ util.verifyRelPlan("SELECT * FROM f(uid => 'my-ptf')");
+ }
+
+ @Test
+ void testPojoArgs() {
+ util.addTemporarySystemFunction("f", PojoArgsFunction.class);
+ util.addTemporarySystemFunction("pojoCreator", PojoCreatingFunction.class);
+ assertReachesOptimizer(
+ "SELECT * FROM f(input => TABLE t1, scalar => pojoCreator('Bob', 12), uid => 'my-ptf')");
+ }
+
+ @Test
+ void testInvalidTableFunction() {
+ util.addTemporarySystemFunction("f", NoProcessTableFunction.class);
+ assertThatThrownBy(() -> util.verifyRelPlan("SELECT * FROM f(r => TABLE t1)"))
+ .hasRootCauseMessage(
+ "Only scalar arguments are supported at this location. "
+ + "But argument 'r' declared the following traits: [TABLE, TABLE_AS_ROW]");
+ }
+
+ @Test
+ void testReservedArg() {
+ util.addTemporarySystemFunction("f", ReservedArgFunction.class);
+ assertThatThrownBy(() -> util.verifyRelPlan("SELECT * FROM f(uid => 'my-ptf')"))
+ .hasRootCauseMessage(
+ "Function signature must not declare system arguments. Reserved argument names are: [uid]");
+ }
Review Comment:
Not sure how easy it is for `success` cases
however at least for `fail` cases wouldn't it make sense to replace them with a couple of parameterized tests?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-36707][table] Integrate PTF type inference into Calcite [flink]
Posted by "twalthr (via GitHub)" <gi...@apache.org>.
twalthr merged PR #25928:
URL: https://github.com/apache/flink/pull/25928
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-36707][table] Integrate PTF type inference into Calcite [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #25928:
URL: https://github.com/apache/flink/pull/25928#discussion_r1907253608
##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/stream/sql/ProcessTableFunctionTest.java:
##########
@@ -0,0 +1,326 @@
+package org.apache.flink.table.planner.plan.stream.sql;
+
+import org.apache.flink.table.annotation.ArgumentHint;
+import org.apache.flink.table.annotation.ArgumentTrait;
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.api.TableConfig;
+import org.apache.flink.table.catalog.DataTypeFactory;
+import org.apache.flink.table.functions.ProcessTableFunction;
+import org.apache.flink.table.functions.ScalarFunction;
+import org.apache.flink.table.functions.TableFunction;
+import org.apache.flink.table.planner.utils.TableTestBase;
+import org.apache.flink.table.planner.utils.TableTestUtil;
+import org.apache.flink.table.types.inference.StaticArgument;
+import org.apache.flink.table.types.inference.StaticArgumentTrait;
+import org.apache.flink.table.types.inference.TypeInference;
+import org.apache.flink.types.Row;
+
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.util.EnumSet;
+import java.util.Optional;
+
+import static org.apache.flink.table.annotation.ArgumentTrait.OPTIONAL_PARTITION_BY;
+import static org.apache.flink.table.annotation.ArgumentTrait.TABLE_AS_ROW;
+import static org.apache.flink.table.annotation.ArgumentTrait.TABLE_AS_SET;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+
+/** Tests for the type inference and planning part of {@link ProcessTableFunction}. */
+public class ProcessTableFunctionTest extends TableTestBase {
+
+ private TableTestUtil util;
+
+ @BeforeEach
+ void setup() {
+ util = streamTestUtil(TableConfig.getDefault());
+ util.tableEnv().executeSql("CREATE VIEW t1 AS SELECT 'Bob' AS name, 12 AS score");
+ util.tableEnv().executeSql("CREATE VIEW t2 AS SELECT 'Bob' AS name, 12 AS different");
+ util.tableEnv().executeSql("CREATE VIEW t3 AS SELECT 'Bob' AS name, TRUE AS isValid");
+ }
+
+ @Test
+ void testScalarArgsNoUid() {
+ util.addTemporarySystemFunction("f", ScalarArgsFunction.class);
+ util.verifyRelPlan("SELECT * FROM f(i => 1, b => true)");
+ }
+
+ @Test
+ void testScalarArgsWithUid() {
+ util.addTemporarySystemFunction("f", ScalarArgsFunction.class);
+ // argument 'uid' is also reordered
+ util.verifyRelPlan("SELECT * FROM f(uid => 'my-uid', i => 1, b => true)");
+ }
+
+ @Test
+ void testUnknownScalarArg() {
+ util.addTemporarySystemFunction("f", ScalarArgsFunction.class);
+ // argument 'invalid' is ignored
+ util.verifyRelPlan("SELECT * FROM f(i => 1, b => true, invalid => 'invalid')");
+ }
+
+ @Test
+ void testInvalidUid() {
+ util.addTemporarySystemFunction("f", ScalarArgsFunction.class);
+ assertThatThrownBy(
+ () -> util.verifyRelPlan("SELECT * FROM f(uid => '%', i => 1, b => true)"))
+ .hasRootCauseMessage(
+ "Invalid unique identifier for process table function. "
+ + "The 'uid' argument must be a string literal that follows the pattern [a-zA-Z_][a-zA-Z-_0-9]*. "
+ + "But found: %");
+ }
+
+ @Test
+ void testTableAsRow() {
+ util.addTemporarySystemFunction("f", TableAsRowFunction.class);
+ assertReachesOptimizer("SELECT * FROM f(r => TABLE t1, i => 1)");
+ }
+
+ @Test
+ void testTypedTableAsRow() {
+ util.addTemporarySystemFunction("f", TypedTableAsRowFunction.class);
+ assertReachesOptimizer("SELECT * FROM f(u => TABLE t1, i => 1)");
+ }
+
+ @Test
+ void testTypedTableAsRowIgnoringColumnNames() {
+ util.addTemporarySystemFunction("f", TypedTableAsRowFunction.class);
+ // function expects <STRING name, INT score>
+ // but table is <STRING name, INT different>
+ assertReachesOptimizer("SELECT * FROM f(u => TABLE t2, i => 1)");
+ }
+
+ @Test
+ void testTypedTableAsRowWithInvalidInput() {
+ util.addTemporarySystemFunction("f", TypedTableAsRowFunction.class);
+ // function expects <STRING name, INT score>
+ assertThatThrownBy(() -> util.verifyRelPlan("SELECT * FROM f(u => TABLE t3, i => 1)"))
+ .hasMessageContaining(
+ "No match found for function signature "
+ + "f(<RecordType(CHAR(3) name, BOOLEAN isValid)>, <NUMERIC>, <CHARACTER>)");
+ }
+
+ @Test
+ void testTableAsSet() {
+ util.addTemporarySystemFunction("f", TableAsSetFunction.class);
+ assertReachesOptimizer("SELECT * FROM f(r => TABLE t1 PARTITION BY name, i => 1)");
+ }
+
+ @Test
+ void testTableAsSetWithInvalidPartitionBy() {
+ util.addTemporarySystemFunction("f", TableAsSetFunction.class);
+ assertThatThrownBy(() -> util.verifyRelPlan("SELECT * FROM f(r => TABLE t1, i => 1)"))
+ .hasRootCauseMessage(
+ "Table argument 'r' requires a PARTITION BY clause for parallel processing.");
+ }
+
+ @Test
+ void testTableAsSetOptionalPartitionBy() {
+ util.addTemporarySystemFunction("f", TableAsSetOptionalPartitionFunction.class);
+ assertReachesOptimizer("SELECT * FROM f(r => TABLE t1, i => 1)");
+ }
+
+ @Test
+ void testTypedTableAsSet() {
+ util.addTemporarySystemFunction("f", TypedTableAsSetFunction.class);
+ assertReachesOptimizer("SELECT * FROM f(u => TABLE t1 PARTITION BY name, i => 1)");
+ }
+
+ @Test
+ void testTypedTableAsSetWithInvalidInput() {
+ util.addTemporarySystemFunction("f", TypedTableAsSetFunction.class);
+ // function expects <STRING name, INT score>
+ assertThatThrownBy(
+ () ->
+ util.verifyRelPlan(
+ "SELECT * FROM f(u => TABLE t3 PARTITION BY name, i => 1)"))
+ .hasMessageContaining(
+ "No match found for function signature "
+ + "f(<RecordType(CHAR(3) name, BOOLEAN isValid)>, <NUMERIC>, <CHARACTER>)");
+ }
+
+ @Test
+ void testEmptyArgs() {
+ util.addTemporarySystemFunction("f", EmptyArgFunction.class);
+ util.verifyRelPlan("SELECT * FROM f(uid => 'my-ptf')");
+ }
+
+ @Test
+ void testPojoArgs() {
+ util.addTemporarySystemFunction("f", PojoArgsFunction.class);
+ util.addTemporarySystemFunction("pojoCreator", PojoCreatingFunction.class);
+ assertReachesOptimizer(
+ "SELECT * FROM f(input => TABLE t1, scalar => pojoCreator('Bob', 12), uid => 'my-ptf')");
+ }
+
+ @Test
+ void testInvalidTableFunction() {
+ util.addTemporarySystemFunction("f", NoProcessTableFunction.class);
+ assertThatThrownBy(() -> util.verifyRelPlan("SELECT * FROM f(r => TABLE t1)"))
+ .hasRootCauseMessage(
+ "Only scalar arguments are supported at this location. "
+ + "But argument 'r' declared the following traits: [TABLE, TABLE_AS_ROW]");
+ }
+
+ @Test
+ void testReservedArg() {
+ util.addTemporarySystemFunction("f", ReservedArgFunction.class);
+ assertThatThrownBy(() -> util.verifyRelPlan("SELECT * FROM f(uid => 'my-ptf')"))
+ .hasRootCauseMessage(
+ "Function signature must not declare system arguments. Reserved argument names are: [uid]");
+ }
+
+ @Test
+ void testInvalidMultiTable() {
+ util.addTemporarySystemFunction("f", MultiTableFunction.class);
+ assertThatThrownBy(
+ () -> util.verifyRelPlan("SELECT * FROM f(r1 => TABLE t1, r2 => TABLE t1)"))
+ .hasRootCauseMessage(
+ "Currently, only signatures with at most one table argument are supported.");
+ }
+
+ @Test
+ void testInvalidRowInsteadOfTable() {
+ util.addTemporarySystemFunction("f", TableAsRowFunction.class);
+ assertThatThrownBy(() -> util.verifyRelPlan("SELECT * FROM f(r => ROW(42), i => 1)"))
+ .hasRootCauseMessage(
+ "Invalid argument value. Argument 'r' expects a table to be passed.");
+ }
+
+ @Test
+ void testInvalidSetSemantics() {
+ util.addTemporarySystemFunction("f", TableAsRowFunction.class);
+ assertThatThrownBy(
+ () ->
+ util.verifyRelPlan(
+ "SELECT * FROM f(r => TABLE t1 PARTITION BY name, i => 1)"))
+ .hasRootCauseMessage(
+ "Only tables with set semantics may be partitioned. "
+ + "Invalid PARTITION BY clause in the 0-th operand of table function 'f'");
+ }
+
+ @Test
+ void testInvalidPartitionByClause() {
+ util.addTemporarySystemFunction("f", TableAsSetFunction.class);
+ assertThatThrownBy(
+ () ->
+ util.verifyRelPlan(
+ "SELECT * FROM f(r => TABLE t1 PARTITION BY invalid, i => 1)"))
+ .hasRootCauseMessage(
+ "Invalid column 'invalid' for PARTITION BY clause. Available columns are: [name, score]");
+ }
+
+ @Test
+ void testUnsupportedOrderByClause() {
+ util.addTemporarySystemFunction("f", TableAsSetFunction.class);
+ assertThatThrownBy(
+ () ->
+ util.verifyRelPlan(
+ "SELECT * FROM f(r => TABLE t1 PARTITION BY name ORDER BY score, i => 1)"))
+ .hasRootCauseMessage("ORDER BY clause is currently not supported.");
+ }
+
+ private void assertReachesOptimizer(String sql) {
+ assertThatThrownBy(() -> util.verifyRelPlan(sql))
+ .hasMessageContaining(
+ "This exception indicates that the query uses an unsupported SQL feature.");
+ }
+
+ /** Testing function. */
Review Comment:
I guess instead of this repeating comment we could just add one like
```java
// -------------------------------------------
// Testing functions
// -------------------------------------------
```
or something like that
WDYT?
##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/stream/sql/ProcessTableFunctionTest.java:
##########
@@ -0,0 +1,326 @@
+package org.apache.flink.table.planner.plan.stream.sql;
+
+import org.apache.flink.table.annotation.ArgumentHint;
+import org.apache.flink.table.annotation.ArgumentTrait;
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.api.TableConfig;
+import org.apache.flink.table.catalog.DataTypeFactory;
+import org.apache.flink.table.functions.ProcessTableFunction;
+import org.apache.flink.table.functions.ScalarFunction;
+import org.apache.flink.table.functions.TableFunction;
+import org.apache.flink.table.planner.utils.TableTestBase;
+import org.apache.flink.table.planner.utils.TableTestUtil;
+import org.apache.flink.table.types.inference.StaticArgument;
+import org.apache.flink.table.types.inference.StaticArgumentTrait;
+import org.apache.flink.table.types.inference.TypeInference;
+import org.apache.flink.types.Row;
+
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.util.EnumSet;
+import java.util.Optional;
+
+import static org.apache.flink.table.annotation.ArgumentTrait.OPTIONAL_PARTITION_BY;
+import static org.apache.flink.table.annotation.ArgumentTrait.TABLE_AS_ROW;
+import static org.apache.flink.table.annotation.ArgumentTrait.TABLE_AS_SET;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+
+/** Tests for the type inference and planning part of {@link ProcessTableFunction}. */
+public class ProcessTableFunctionTest extends TableTestBase {
+
+ private TableTestUtil util;
+
+ @BeforeEach
+ void setup() {
+ util = streamTestUtil(TableConfig.getDefault());
+ util.tableEnv().executeSql("CREATE VIEW t1 AS SELECT 'Bob' AS name, 12 AS score");
+ util.tableEnv().executeSql("CREATE VIEW t2 AS SELECT 'Bob' AS name, 12 AS different");
+ util.tableEnv().executeSql("CREATE VIEW t3 AS SELECT 'Bob' AS name, TRUE AS isValid");
+ }
+
+ @Test
+ void testScalarArgsNoUid() {
+ util.addTemporarySystemFunction("f", ScalarArgsFunction.class);
+ util.verifyRelPlan("SELECT * FROM f(i => 1, b => true)");
+ }
+
+ @Test
+ void testScalarArgsWithUid() {
+ util.addTemporarySystemFunction("f", ScalarArgsFunction.class);
+ // argument 'uid' is also reordered
+ util.verifyRelPlan("SELECT * FROM f(uid => 'my-uid', i => 1, b => true)");
+ }
+
+ @Test
+ void testUnknownScalarArg() {
+ util.addTemporarySystemFunction("f", ScalarArgsFunction.class);
+ // argument 'invalid' is ignored
+ util.verifyRelPlan("SELECT * FROM f(i => 1, b => true, invalid => 'invalid')");
+ }
+
+ @Test
+ void testInvalidUid() {
+ util.addTemporarySystemFunction("f", ScalarArgsFunction.class);
+ assertThatThrownBy(
+ () -> util.verifyRelPlan("SELECT * FROM f(uid => '%', i => 1, b => true)"))
+ .hasRootCauseMessage(
+ "Invalid unique identifier for process table function. "
+ + "The 'uid' argument must be a string literal that follows the pattern [a-zA-Z_][a-zA-Z-_0-9]*. "
+ + "But found: %");
+ }
+
+ @Test
+ void testTableAsRow() {
+ util.addTemporarySystemFunction("f", TableAsRowFunction.class);
+ assertReachesOptimizer("SELECT * FROM f(r => TABLE t1, i => 1)");
+ }
+
+ @Test
+ void testTypedTableAsRow() {
+ util.addTemporarySystemFunction("f", TypedTableAsRowFunction.class);
+ assertReachesOptimizer("SELECT * FROM f(u => TABLE t1, i => 1)");
+ }
+
+ @Test
+ void testTypedTableAsRowIgnoringColumnNames() {
+ util.addTemporarySystemFunction("f", TypedTableAsRowFunction.class);
+ // function expects <STRING name, INT score>
+ // but table is <STRING name, INT different>
+ assertReachesOptimizer("SELECT * FROM f(u => TABLE t2, i => 1)");
+ }
+
+ @Test
+ void testTypedTableAsRowWithInvalidInput() {
+ util.addTemporarySystemFunction("f", TypedTableAsRowFunction.class);
+ // function expects <STRING name, INT score>
+ assertThatThrownBy(() -> util.verifyRelPlan("SELECT * FROM f(u => TABLE t3, i => 1)"))
+ .hasMessageContaining(
+ "No match found for function signature "
+ + "f(<RecordType(CHAR(3) name, BOOLEAN isValid)>, <NUMERIC>, <CHARACTER>)");
+ }
+
+ @Test
+ void testTableAsSet() {
+ util.addTemporarySystemFunction("f", TableAsSetFunction.class);
+ assertReachesOptimizer("SELECT * FROM f(r => TABLE t1 PARTITION BY name, i => 1)");
+ }
+
+ @Test
+ void testTableAsSetWithInvalidPartitionBy() {
+ util.addTemporarySystemFunction("f", TableAsSetFunction.class);
+ assertThatThrownBy(() -> util.verifyRelPlan("SELECT * FROM f(r => TABLE t1, i => 1)"))
+ .hasRootCauseMessage(
+ "Table argument 'r' requires a PARTITION BY clause for parallel processing.");
+ }
+
+ @Test
+ void testTableAsSetOptionalPartitionBy() {
+ util.addTemporarySystemFunction("f", TableAsSetOptionalPartitionFunction.class);
+ assertReachesOptimizer("SELECT * FROM f(r => TABLE t1, i => 1)");
+ }
+
+ @Test
+ void testTypedTableAsSet() {
+ util.addTemporarySystemFunction("f", TypedTableAsSetFunction.class);
+ assertReachesOptimizer("SELECT * FROM f(u => TABLE t1 PARTITION BY name, i => 1)");
+ }
+
+ @Test
+ void testTypedTableAsSetWithInvalidInput() {
+ util.addTemporarySystemFunction("f", TypedTableAsSetFunction.class);
+ // function expects <STRING name, INT score>
+ assertThatThrownBy(
+ () ->
+ util.verifyRelPlan(
+ "SELECT * FROM f(u => TABLE t3 PARTITION BY name, i => 1)"))
+ .hasMessageContaining(
+ "No match found for function signature "
+ + "f(<RecordType(CHAR(3) name, BOOLEAN isValid)>, <NUMERIC>, <CHARACTER>)");
+ }
+
+ @Test
+ void testEmptyArgs() {
+ util.addTemporarySystemFunction("f", EmptyArgFunction.class);
+ util.verifyRelPlan("SELECT * FROM f(uid => 'my-ptf')");
+ }
+
+ @Test
+ void testPojoArgs() {
+ util.addTemporarySystemFunction("f", PojoArgsFunction.class);
+ util.addTemporarySystemFunction("pojoCreator", PojoCreatingFunction.class);
+ assertReachesOptimizer(
+ "SELECT * FROM f(input => TABLE t1, scalar => pojoCreator('Bob', 12), uid => 'my-ptf')");
+ }
+
+ @Test
+ void testInvalidTableFunction() {
+ util.addTemporarySystemFunction("f", NoProcessTableFunction.class);
+ assertThatThrownBy(() -> util.verifyRelPlan("SELECT * FROM f(r => TABLE t1)"))
+ .hasRootCauseMessage(
+ "Only scalar arguments are supported at this location. "
+ + "But argument 'r' declared the following traits: [TABLE, TABLE_AS_ROW]");
+ }
+
+ @Test
+ void testReservedArg() {
+ util.addTemporarySystemFunction("f", ReservedArgFunction.class);
+ assertThatThrownBy(() -> util.verifyRelPlan("SELECT * FROM f(uid => 'my-ptf')"))
+ .hasRootCauseMessage(
+ "Function signature must not declare system arguments. Reserved argument names are: [uid]");
+ }
+
+ @Test
+ void testInvalidMultiTable() {
+ util.addTemporarySystemFunction("f", MultiTableFunction.class);
+ assertThatThrownBy(
+ () -> util.verifyRelPlan("SELECT * FROM f(r1 => TABLE t1, r2 => TABLE t1)"))
+ .hasRootCauseMessage(
+ "Currently, only signatures with at most one table argument are supported.");
+ }
+
+ @Test
+ void testInvalidRowInsteadOfTable() {
+ util.addTemporarySystemFunction("f", TableAsRowFunction.class);
+ assertThatThrownBy(() -> util.verifyRelPlan("SELECT * FROM f(r => ROW(42), i => 1)"))
+ .hasRootCauseMessage(
+ "Invalid argument value. Argument 'r' expects a table to be passed.");
+ }
+
+ @Test
+ void testInvalidSetSemantics() {
+ util.addTemporarySystemFunction("f", TableAsRowFunction.class);
+ assertThatThrownBy(
+ () ->
+ util.verifyRelPlan(
+ "SELECT * FROM f(r => TABLE t1 PARTITION BY name, i => 1)"))
+ .hasRootCauseMessage(
+ "Only tables with set semantics may be partitioned. "
+ + "Invalid PARTITION BY clause in the 0-th operand of table function 'f'");
+ }
+
+ @Test
+ void testInvalidPartitionByClause() {
+ util.addTemporarySystemFunction("f", TableAsSetFunction.class);
+ assertThatThrownBy(
+ () ->
+ util.verifyRelPlan(
+ "SELECT * FROM f(r => TABLE t1 PARTITION BY invalid, i => 1)"))
+ .hasRootCauseMessage(
+ "Invalid column 'invalid' for PARTITION BY clause. Available columns are: [name, score]");
+ }
+
+ @Test
+ void testUnsupportedOrderByClause() {
+ util.addTemporarySystemFunction("f", TableAsSetFunction.class);
+ assertThatThrownBy(
+ () ->
+ util.verifyRelPlan(
+ "SELECT * FROM f(r => TABLE t1 PARTITION BY name ORDER BY score, i => 1)"))
+ .hasRootCauseMessage("ORDER BY clause is currently not supported.");
+ }
+
+ private void assertReachesOptimizer(String sql) {
+ assertThatThrownBy(() -> util.verifyRelPlan(sql))
+ .hasMessageContaining(
+ "This exception indicates that the query uses an unsupported SQL feature.");
+ }
+
+ /** Testing function. */
Review Comment:
I guess instead of this repeating comment we could just add one like
```java
// -------------------------------------------
// Testing functions
// -------------------------------------------
```
or something like that
WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-36707][table] Integrate PTF type inference into Calcite [flink]
Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #25928:
URL: https://github.com/apache/flink/pull/25928#issuecomment-2577802374
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "14b33286f0e1c88f687bb1edf244dc7f9cb8c1a7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "14b33286f0e1c88f687bb1edf244dc7f9cb8c1a7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 14b33286f0e1c88f687bb1edf244dc7f9cb8c1a7 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org