You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2022/05/26 13:17:37 UTC
[GitHub] [flink] luoyuxia commented on a diff in pull request #19742: [FLINK-27651][table] Support CREATE FUNCTION USING JAR syntax
luoyuxia commented on code in PR #19742:
URL: https://github.com/apache/flink/pull/19742#discussion_r882640831
##########
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/catalog/CatalogFunctionImpl.java:
##########
@@ -50,7 +60,7 @@ public String getClassName() {
@Override
public CatalogFunction copy() {
- return new CatalogFunctionImpl(getClassName(), functionLanguage);
+ return new CatalogFunctionImpl(getClassName(), functionLanguage, resourceUris);
Review Comment:
Is it expected that the copy isnot deep copy?
##########
flink-table/flink-sql-parser/src/main/java/org/apache/flink/sql/parser/ddl/SqlCreateFunction.java:
##########
@@ -51,20 +53,24 @@ public class SqlCreateFunction extends SqlCreate {
private final boolean isSystemFunction;
+ private final SqlNodeList jarPaths;
Review Comment:
I suggest to rename it for it may be a file instead of a jar, althogh currently we only support jar.
##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/operations/SqlToOperationConverterTest.java:
##########
@@ -1213,6 +1218,54 @@ public void testCreateTableWithMetadataColumn() {
assertThat(actualSchema).isEqualTo(expectedSchema);
}
+ @Test
+ public void testCreateFunction() {
+ // test create catalog function
+ String sql =
+ "CREATE FUNCTION test_udf AS 'org.apache.fink.function.function1' "
+ + "LANGUAGE JAVA USING JAR 'file:///path/to/test.jar'";
+ final FlinkPlannerImpl planner = getPlannerBySqlDialect(SqlDialect.DEFAULT);
+ Operation operation = parse(sql, planner, getParserBySqlDialect(SqlDialect.DEFAULT));
+ assertThat(operation).isInstanceOf(CreateCatalogFunctionOperation.class);
+ CatalogFunction actualFunction =
+ ((CreateCatalogFunctionOperation) operation).getCatalogFunction();
+
+ assertThat(operation.asSummaryString())
+ .isEqualTo(
+ "CREATE CATALOG FUNCTION: (catalogFunction: [Optional[This is a user-defined function]], "
+ + "identifier: [`builtin`.`default`.`test_udf`], ignoreIfExists: [false], isTemporary: [false])");
+
+ // here doesn't assert the CatalogFunction directly because of the isGeneric method will
+ // load the class
+ assertThat(actualFunction.getClassName()).isEqualTo("org.apache.fink.function.function1");
+ assertThat(actualFunction.getFunctionLanguage()).isEqualTo(FunctionLanguage.JAVA);
+ assertThat(actualFunction.getFunctionResources())
+ .isEqualTo(
+ Arrays.asList(
+ new ResourceUri(ResourceType.JAR, "file:///path/to/test.jar")));
+
+ // test create temporary system function
+ sql =
+ "CREATE TEMPORARY SYSTEM FUNCTION test_udf2 AS 'org.apache.fink.function.function2' "
+ + "LANGUAGE SCALA USING JAR 'file:///path/to/test.jar'";
+ operation = parse(sql, planner, getParserBySqlDialect(SqlDialect.DEFAULT));
+ assertThat(operation).isInstanceOf(CreateTempSystemFunctionOperation.class);
+ CreateTempSystemFunctionOperation tempSystemFunctionOperation =
+ (CreateTempSystemFunctionOperation) operation;
+ actualFunction = tempSystemFunctionOperation.getCatalogFunction();
+
+ // here doesn't assert the CreateTempSystemFunctionOperation directly because of the
+ // isGeneric method will load the class
+ assertThat(tempSystemFunctionOperation.getFunctionName()).isEqualTo("test_udf2");
+ assertThat(tempSystemFunctionOperation.isIgnoreIfExists()).isEqualTo(false);
+ assertThat(actualFunction.getClassName()).isEqualTo("org.apache.fink.function.function2");
+ assertThat(actualFunction.getFunctionLanguage()).isEqualTo(FunctionLanguage.SCALA);
+ assertThat(actualFunction.getFunctionResources())
+ .isEqualTo(
+ Arrays.asList(
Review Comment:
nit:
Collections.singletonList(
##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/operations/SqlToOperationConverterTest.java:
##########
@@ -1213,6 +1218,54 @@ public void testCreateTableWithMetadataColumn() {
assertThat(actualSchema).isEqualTo(expectedSchema);
}
+ @Test
+ public void testCreateFunction() {
+ // test create catalog function
+ String sql =
+ "CREATE FUNCTION test_udf AS 'org.apache.fink.function.function1' "
+ + "LANGUAGE JAVA USING JAR 'file:///path/to/test.jar'";
+ final FlinkPlannerImpl planner = getPlannerBySqlDialect(SqlDialect.DEFAULT);
+ Operation operation = parse(sql, planner, getParserBySqlDialect(SqlDialect.DEFAULT));
+ assertThat(operation).isInstanceOf(CreateCatalogFunctionOperation.class);
+ CatalogFunction actualFunction =
+ ((CreateCatalogFunctionOperation) operation).getCatalogFunction();
+
+ assertThat(operation.asSummaryString())
+ .isEqualTo(
+ "CREATE CATALOG FUNCTION: (catalogFunction: [Optional[This is a user-defined function]], "
+ + "identifier: [`builtin`.`default`.`test_udf`], ignoreIfExists: [false], isTemporary: [false])");
+
+ // here doesn't assert the CatalogFunction directly because of the isGeneric method will
+ // load the class
+ assertThat(actualFunction.getClassName()).isEqualTo("org.apache.fink.function.function1");
+ assertThat(actualFunction.getFunctionLanguage()).isEqualTo(FunctionLanguage.JAVA);
+ assertThat(actualFunction.getFunctionResources())
+ .isEqualTo(
+ Arrays.asList(
Review Comment:
nit:
Collections.singletonList(
##########
flink-table/flink-sql-parser/src/main/java/org/apache/flink/sql/parser/ddl/SqlCreateFunction.java:
##########
@@ -123,4 +139,11 @@ public String getFunctionLanguage() {
public String[] getFunctionIdentifier() {
return functionIdentifier.names.toArray(new String[0]);
}
+
+ public List<String> getJarPaths() {
Review Comment:
Also suggest to rename this method.
##########
flink-table/flink-sql-parser/src/test/java/org/apache/flink/sql/parser/FlinkSqlParserImplTest.java:
##########
@@ -1314,6 +1314,18 @@ void testCreateFunction() {
"CREATE SYSTEM FUNCTION is not supported, "
+ "system functions can only be registered as temporary "
+ "function, you can use CREATE TEMPORARY SYSTEM FUNCTION instead.");
+
+ // test create function using jar
+ sql("create temporary function function1 as 'org.apache.fink.function.function1' language java using jar 'file:///path/to/test.jar'")
+ .ok(
+ "CREATE TEMPORARY FUNCTION `FUNCTION1` AS 'org.apache.fink.function.function1' LANGUAGE JAVA USING JAR 'file:///path/to/test.jar'");
+
+ sql("create function function1 as 'org.apache.fink.function.function1' language java using jar 'file:///path/to/test.jar', jar 'hdfs:///path/to/test2.jar'")
Review Comment:
add test for `LANGUAGE SCALA/PYTHON`
##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/operations/SqlToOperationConverter.java:
##########
@@ -652,19 +654,24 @@ private ModifyOperation convertAlterTableCompact(
private Operation convertCreateFunction(SqlCreateFunction sqlCreateFunction) {
UnresolvedIdentifier unresolvedIdentifier =
UnresolvedIdentifier.of(sqlCreateFunction.getFunctionIdentifier());
-
+ List<ResourceUri> resourceUris =
+ sqlCreateFunction.getJarPaths().stream()
+ .map(path -> new ResourceUri(ResourceType.JAR, path))
Review Comment:
It'll always create jar resource. It's fine when we only support jar resource. But as we want to support other resources in the future, here seems is not generic.
Some guys may struggle to modify it when them want to support other resources.
##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/operations/SqlToOperationConverterTest.java:
##########
@@ -1213,6 +1218,54 @@ public void testCreateTableWithMetadataColumn() {
assertThat(actualSchema).isEqualTo(expectedSchema);
}
+ @Test
+ public void testCreateFunction() {
+ // test create catalog function
+ String sql =
+ "CREATE FUNCTION test_udf AS 'org.apache.fink.function.function1' "
+ + "LANGUAGE JAVA USING JAR 'file:///path/to/test.jar'";
+ final FlinkPlannerImpl planner = getPlannerBySqlDialect(SqlDialect.DEFAULT);
+ Operation operation = parse(sql, planner, getParserBySqlDialect(SqlDialect.DEFAULT));
+ assertThat(operation).isInstanceOf(CreateCatalogFunctionOperation.class);
+ CatalogFunction actualFunction =
+ ((CreateCatalogFunctionOperation) operation).getCatalogFunction();
+
+ assertThat(operation.asSummaryString())
+ .isEqualTo(
+ "CREATE CATALOG FUNCTION: (catalogFunction: [Optional[This is a user-defined function]], "
+ + "identifier: [`builtin`.`default`.`test_udf`], ignoreIfExists: [false], isTemporary: [false])");
+
+ // here doesn't assert the CatalogFunction directly because of the isGeneric method will
+ // load the class
+ assertThat(actualFunction.getClassName()).isEqualTo("org.apache.fink.function.function1");
Review Comment:
how about extract the assert logic to a common method so that the following assert can reuse it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org