You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "yuxiqian (via GitHub)" <gi...@apache.org> on 2024/07/10 09:27:27 UTC
[PR] [FLINK-34876][transform] Support UDF functions in transform [flink-cdc]
yuxiqian opened a new pull request, #3465:
URL: https://github.com/apache/flink-cdc/pull/3465
This closes FLINK-34876.
By allowing providing UDF function classpath, users may provide self-defined scalar functions and use them in transform projection / filtering rules.
This is a primitive implementation, eagerly looking for any advice.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34876][transform] Support UDF functions in transform [flink-cdc]
Posted by "yuxiqian (via GitHub)" <gi...@apache.org>.
yuxiqian commented on PR #3465:
URL: https://github.com/apache/flink-cdc/pull/3465#issuecomment-2262911364
Adjusted based on previous discussions, cleaned up changes, and added UDF-related docs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34876][transform] Support UDF functions in transform [flink-cdc]
Posted by "yuxiqian (via GitHub)" <gi...@apache.org>.
yuxiqian commented on PR #3465:
URL: https://github.com/apache/flink-cdc/pull/3465#issuecomment-2260265325
Updated API design & POC implementation, could @aiwenmo please take a look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34876][transform] Support UDF functions in transform [flink-cdc]
Posted by "yuxiqian (via GitHub)" <gi...@apache.org>.
yuxiqian commented on PR #3465:
URL: https://github.com/apache/flink-cdc/pull/3465#issuecomment-2275118059
Rebased to latest `master`, cc @ruanhang1993
Minor concern: By adding more E2e test cases, now it takes nearly 1h 30min to finish running e2e case, which is worrying since current timeout limit is configured to 90 minutes. Maybe something like #3514 is necessary?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34876][transform] Support UDF functions in transform [flink-cdc]
Posted by "ruanhang1993 (via GitHub)" <gi...@apache.org>.
ruanhang1993 commented on code in PR #3465:
URL: https://github.com/apache/flink-cdc/pull/3465#discussion_r1708813058
##########
docs/content.zh/docs/core-concept/transform.md:
##########
@@ -266,6 +266,75 @@ transform:
description: classification mapping example
```
+## User-defined Functions
+
+User-defined functions (UDFs) can be used in transform rules.
+
+Classes could be used as a UDF if:
+
+* implements `org.apache.flink.cdc.common.udf.UserDefinedFunction` interface
+* has a public constructor with no parameters
+* has at least one public method named `eval`
+
+It may also:
+
+* overrides `getReturnType` method to indicate its return CDC type
+* overrides `open` and `close` method to do some initialization and cleanup work
+
+For example, this is a valid UDF class:
+
+```java
+public class AddOneFunctionClass implements UserDefinedFunction {
+
+ public Object eval(Integer num) {
+ return num + 1;
+ }
+
+ @Override
+ public DataType getReturnType() {
+ return DataTypes.INT();
+ }
+
+ @Override
+ public void open() throws Exception {
+ // ...
+ }
+
+ @Override
+ public void close() throws Exception {
+ // ...
+ }
+}
+```
+
+To ease the migration from Flink SQL to Flink CDC, a Flink `ScalarFunction` could also be used as a transform UDF, with some limitations:
+
+* `ScalarFunction` with parameters is not supported.
Review Comment:
`ScalarFunction` which has a constructor with parameters is not supported.
##########
flink-cdc-cli/src/main/java/org/apache/flink/cdc/cli/parser/YamlPipelineDefinitionParser.java:
##########
@@ -199,6 +211,23 @@ private RouteDef toRouteDef(JsonNode routeNode) {
return new RouteDef(sourceTable, sinkTable, replaceSymbol, description);
}
+ private UdfDef toUdfDef(JsonNode udfNode) {
+ String functionName =
+ checkNotNull(
+ udfNode.get(UDF_FUNCTION_NAME_KEY),
+ "Missing required field \"%s\" in UDF configuration",
+ UDF_FUNCTION_NAME_KEY)
+ .asText();
+ String classPath =
Review Comment:
```suggestion
String classpath =
```
##########
flink-cdc-composer/src/main/java/org/apache/flink/cdc/composer/flink/FlinkPipelineComposer.java:
##########
@@ -96,6 +98,11 @@ public PipelineExecution compose(PipelineDef pipelineDef) {
int parallelism = pipelineDef.getConfig().get(PipelineOptions.PIPELINE_PARALLELISM);
env.getConfig().setParallelism(parallelism);
+ List<Tuple2<String, String>> udfFunctions =
+ pipelineDef.getUdfs().stream()
+ .map(udf -> Tuple2.of(udf.getName(), udf.getClassPath()))
+ .collect(Collectors.toList());
Review Comment:
This translation should be placed in the `transformTranslator`.
##########
flink-cdc-composer/src/main/java/org/apache/flink/cdc/composer/flink/FlinkPipelineComposer.java:
##########
@@ -106,7 +113,9 @@ public PipelineExecution compose(PipelineDef pipelineDef) {
// Build TransformSchemaOperator for processing Schema Event
TransformTranslator transformTranslator = new TransformTranslator();
- stream = transformTranslator.translateSchema(stream, pipelineDef.getTransforms());
+ stream =
+ transformTranslator.translateSchema(
+ stream, pipelineDef.getTransforms(), udfFunctions);
Review Comment:
```suggestion
stream, pipelineDef.getTransforms(), pipelineDef.getUdfs());
```
##########
flink-cdc-composer/src/main/java/org/apache/flink/cdc/composer/flink/FlinkPipelineComposer.java:
##########
@@ -125,7 +134,8 @@ public PipelineExecution compose(PipelineDef pipelineDef) {
stream,
pipelineDef.getTransforms(),
schemaOperatorIDGenerator.generate(),
- pipelineDef.getConfig().get(PipelineOptions.PIPELINE_LOCAL_TIME_ZONE));
+ pipelineDef.getConfig().get(PipelineOptions.PIPELINE_LOCAL_TIME_ZONE),
+ udfFunctions);
Review Comment:
```suggestion
pipelineDef.getUdfs());
```
##########
flink-cdc-composer/src/main/java/org/apache/flink/cdc/composer/definition/UdfDef.java:
##########
@@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.cdc.composer.definition;
+
+import java.util.Objects;
+
+/**
+ * Definition of a transformation.
Review Comment:
```suggestion
* Definition of a user-defined function.
```
##########
flink-cdc-pipeline-udf-examples/pom.xml:
##########
@@ -0,0 +1,104 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+ <parent>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-cdc</artifactId>
+ <version>${revision}</version>
+ </parent>
+
+ <artifactId>flink-cdc-pipeline-udf-examples</artifactId>
+
+ <properties>
+ <maven.compiler.source>8</maven.compiler.source>
+ <maven.compiler.target>8</maven.compiler.target>
+ <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
+ <scala.plugin.version>4.9.2</scala.plugin.version>
+ <compiler.encoding>UTF-8</compiler.encoding>
+ </properties>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-cdc-common</artifactId>
+ <version>${project.version}</version>
+ <scope>compile</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.scala-lang</groupId>
+ <artifactId>scala-library</artifactId>
+ <version>${scala.version}</version>
+ </dependency>
+ </dependencies>
+
+ <build>
+ <plugins>
+
+ <!-- other plugins ... -->
+
+ <!-- set scala plugin before compiler plugin -->
+ <plugin>
+ <groupId>net.alchim31.maven</groupId>
+ <artifactId>scala-maven-plugin</artifactId>
+ <version>${scala.plugin.version}</version>
+ <configuration>
+ <recompileMode>incremental</recompileMode>
+ <javacArgs>
+ <javacArg>-Xlint:unchecked</javacArg>
+ <javacArg>-Xlint:deprecation</javacArg>
+ <javacArg>-encoding</javacArg>
+ <javacArg>${compiler.encoding}</javacArg>
+ </javacArgs>
+ </configuration>
+ <executions>
+ <execution>
+ <id>scala-compile-first</id>
+ <phase>process-resources</phase>
+ <goals>
+ <goal>add-source</goal>
+ <goal>compile</goal>
+ </goals>
+ </execution>
+ <execution>
+ <id>scala-test-compile</id>
+ <phase>process-test-resources</phase>
+ <goals>
+ <goal>testCompile</goal>
+ </goals>
+ </execution>
+ </executions>
+ </plugin>
+
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-compiler-plugin</artifactId>
+ <executions>
+ <execution>
+ <phase>compile</phase>
+ <goals>
+ <goal>compile</goal>
+ </goals>
+ </execution>
+ </executions>
+ </plugin>
+
+ </plugins>
+ </build>
+</project>
Review Comment:
Add an empty line.
##########
flink-cdc-common/src/main/java/org/apache/flink/cdc/common/udf/UserDefinedFunction.java:
##########
@@ -0,0 +1,38 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.cdc.common.udf;
+
+import org.apache.flink.cdc.common.annotation.PublicEvolving;
+import org.apache.flink.cdc.common.types.DataType;
+
+/**
+ * Base interface for creating a UDF in transform projection and filtering expressions. You should
+ * define at least one {@code eval} method.
+ */
+@PublicEvolving
+public interface UserDefinedFunction {
+ default DataType getReturnType() {
+ return null;
+ }
Review Comment:
```suggestion
default Optional<DataType> getReturnType() {
return Optional.empty();
}
```
##########
flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/transform/UserDefinedFunctionDescriptor.java:
##########
@@ -0,0 +1,138 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.cdc.runtime.operators.transform;
+
+import org.apache.flink.cdc.common.annotation.Internal;
+import org.apache.flink.cdc.common.types.DataType;
+import org.apache.flink.cdc.common.udf.UserDefinedFunction;
+
+import java.io.Serializable;
+import java.lang.reflect.InvocationTargetException;
+import java.util.Arrays;
+import java.util.Objects;
+import java.util.stream.Collectors;
+
+/** Descriptor of a UDF function. */
+@Internal
+public class UserDefinedFunctionDescriptor implements Serializable {
+
+ private static final long serialVersionUID = 1L;
+
+ private final String name;
+ private final String classPath;
Review Comment:
```suggestion
private final String classpath;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34876][transform] Support UDF functions in transform [flink-cdc]
Posted by "yuxiqian (via GitHub)" <gi...@apache.org>.
yuxiqian commented on code in PR #3465:
URL: https://github.com/apache/flink-cdc/pull/3465#discussion_r1708908351
##########
flink-cdc-common/src/main/java/org/apache/flink/cdc/common/udf/UserDefinedFunction.java:
##########
@@ -0,0 +1,38 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.cdc.common.udf;
+
+import org.apache.flink.cdc.common.annotation.PublicEvolving;
+import org.apache.flink.cdc.common.types.DataType;
+
+/**
+ * Base interface for creating a UDF in transform projection and filtering expressions. You should
+ * define at least one {@code eval} method.
+ */
+@PublicEvolving
+public interface UserDefinedFunction {
+ default DataType getReturnType() {
+ return null;
+ }
Review Comment:
I have some doubt on this. This is meant to be a interface protocol, and any reasonable overwrite of this method should not return an empty value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34876][transform] Support UDF functions in transform [flink-cdc]
Posted by "mamineturki (via GitHub)" <gi...@apache.org>.
mamineturki commented on PR #3465:
URL: https://github.com/apache/flink-cdc/pull/3465#issuecomment-2374250013
Thanks for the speedy response @yuxiqian . passing the jar using `--jar` worked for me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34876][transform] Support UDF functions in transform [flink-cdc]
Posted by "ruanhang1993 (via GitHub)" <gi...@apache.org>.
ruanhang1993 commented on code in PR #3465:
URL: https://github.com/apache/flink-cdc/pull/3465#discussion_r1694814738
##########
flink-cdc-runtime/src/test/java/org/apache/flink/cdc/runtime/parser/JaninoCompilerTest.java:
##########
@@ -118,7 +119,7 @@ public void testJaninoTimestampFunction() throws InvocationTargetException {
List<Object> params = Arrays.asList(epochTime);
ExpressionEvaluator expressionEvaluator =
JaninoCompiler.compileExpression(
- JaninoCompiler.loadSystemFunction(expression),
Review Comment:
Please add some unit tests and e2e tests about the udf.
##########
docs/content/docs/core-concept/data-pipeline.md:
##########
@@ -79,13 +79,30 @@ We could use following yaml file to define a complicated Data Pipeline describin
fenodes: 127.0.0.1:8030
username: root
password: ""
+
+ transform:
+ - source-table: adb.web_order01
+ projection: \*, UPPER(product_name) as product_name
+ filter: id > 10 AND order_id > 100
+ description: project fields and filter
+ - source-table: adb.web_order02
+ projection: \*, UPPER(product_name) as product_name
+ filter: id > 20 AND order_id > 200
+ description: project fields and filter
+
route:
- source-table: app_db.orders
sink-table: ods_db.ods_orders
- source-table: app_db.shipments
sink-table: ods_db.ods_shipments
- source-table: app_db.products
- sink-table: ods_db.ods_products
+ sink-table: ods_db.ods_products
+
+ udf:
+ - name: substring
+ classpath: com.example.functions.scalar.SubStringFunctionClass
Review Comment:
Please add SubStringFunctionClass's code in the doc. And describe how to append them in the classpath.
##########
docs/content/docs/core-concept/data-pipeline.md:
##########
@@ -79,13 +79,30 @@ We could use following yaml file to define a complicated Data Pipeline describin
fenodes: 127.0.0.1:8030
username: root
password: ""
+
+ transform:
+ - source-table: adb.web_order01
+ projection: \*, UPPER(product_name) as product_name
+ filter: id > 10 AND order_id > 100
+ description: project fields and filter
+ - source-table: adb.web_order02
+ projection: \*, UPPER(product_name) as product_name
+ filter: id > 20 AND order_id > 200
+ description: project fields and filter
+
route:
- source-table: app_db.orders
sink-table: ods_db.ods_orders
- source-table: app_db.shipments
sink-table: ods_db.ods_shipments
- source-table: app_db.products
- sink-table: ods_db.ods_products
+ sink-table: ods_db.ods_products
+
+ udf:
+ - name: substring
Review Comment:
Add the usage of this udf in this pipeline.
##########
flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/parser/TransformParser.java:
##########
@@ -87,15 +96,34 @@ private static SqlParser getCalciteParser(String sql) {
.withLex(Lex.JAVA));
}
- private static RelNode sqlToRel(List<Column> columns, SqlNode sqlNode) {
+ private static RelNode sqlToRel(
+ List<Column> columns, SqlNode sqlNode, List<Tuple2<String, String>> udfs) {
List<Column> columnsWithMetadata = copyFillMetadataColumn(sqlNode.toString(), columns);
CalciteSchema rootSchema = CalciteSchema.createRootSchema(true);
+ SchemaPlus schema = rootSchema.plus();
Map<String, Object> operand = new HashMap<>();
operand.put("tableName", DEFAULT_TABLE);
operand.put("columns", columnsWithMetadata);
rootSchema.add(
DEFAULT_SCHEMA,
- TransformSchemaFactory.INSTANCE.create(rootSchema.plus(), DEFAULT_SCHEMA, operand));
+ TransformSchemaFactory.INSTANCE.create(schema, DEFAULT_SCHEMA, operand));
+ List<SqlFunction> udfFunctions = new ArrayList<>();
+ for (Tuple2<String, String> udf : udfs) {
+ try {
+ ScalarFunction function = ScalarFunctionImpl.create(Class.forName(udf.f1), udf.f0);
Review Comment:
Could users use their existed flink sql's scalar function here? I think we should support the flink sql udfs to reduce the difficulty of user migration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34876][transform] Support UDF functions in transform [flink-cdc]
Posted by "yuxiqian (via GitHub)" <gi...@apache.org>.
yuxiqian commented on PR #3465:
URL: https://github.com/apache/flink-cdc/pull/3465#issuecomment-2220009651
Having some difficulties getting Calcite related materials, looking for @aiwenmo's comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34876][transform] Support UDF functions in transform [flink-cdc]
Posted by "mamineturki (via GitHub)" <gi...@apache.org>.
mamineturki commented on PR #3465:
URL: https://github.com/apache/flink-cdc/pull/3465#issuecomment-2374243091
Thanks for the speedy reply @yuxiqian . moving the jar to `Flink lib` did not help. passing the jar with `--jar` caused the `Flink-cdc.sh` command to get stuck with no logs, and no submitted job.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34876][transform] Support UDF functions in transform [flink-cdc]
Posted by "mamineturki (via GitHub)" <gi...@apache.org>.
mamineturki commented on PR #3465:
URL: https://github.com/apache/flink-cdc/pull/3465#issuecomment-2374187978
@yuxiqian Hi, I have been trying to run an example with this new feature but I always get `java.lang.ClassNotFoundException: org.example.udf.AddOneFunctionClass`
I tried loading the udf function using the path in the PR description, as well as the package name `org.apache.flink.cdc.udf.examples.scala.AddOneFunctionClass` with no luck.
I imported the function by downloading the pre-packaged jar `wget https://repo1.maven.org/maven2/org/apache/flink/flink-cdc-pipeline-udf-examples/3.2.0/flink-cdc-pipeline-udf-examples-3.2.0.jar`
please advise on what am I missing exactly. Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34876][transform] Support UDF functions in transform [flink-cdc]
Posted by "yuxiqian (via GitHub)" <gi...@apache.org>.
yuxiqian commented on PR #3465:
URL: https://github.com/apache/flink-cdc/pull/3465#issuecomment-2374215861
Hi @mamineturki, `flink-cdc-pipeline-udf-examples` are for testing purposes only. If you'd like to test it, using `org.apache.flink.cdc.udf.examples.scala.AddOneFunctionClass` should work. Seems it wasn't successfully loaded by Flink classloaders in your case.
Have you tried putting `.jar` file into Flink's `/lib` path (not Flink CDC `/lib`!) and restart the Flink cluster? If that doesn't work, try passing it with `--jar XXX.jar` argument when submitting pipeline job with `bin/flink-cdc.sh`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34876][transform] Support UDF functions in transform [flink-cdc]
Posted by "ruanhang1993 (via GitHub)" <gi...@apache.org>.
ruanhang1993 merged PR #3465:
URL: https://github.com/apache/flink-cdc/pull/3465
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34876][transform] Support UDF functions in transform [flink-cdc]
Posted by "yuxiqian (via GitHub)" <gi...@apache.org>.
yuxiqian commented on PR #3465:
URL: https://github.com/apache/flink-cdc/pull/3465#issuecomment-2275525755
Seems CI is failing due to an expired link in Doris docs. Rebased & fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34876][transform] Support UDF functions in transform [flink-cdc]
Posted by "yuxiqian (via GitHub)" <gi...@apache.org>.
yuxiqian commented on PR #3465:
URL: https://github.com/apache/flink-cdc/pull/3465#issuecomment-2277077343
Rebased with `master` and resolved conflicts. cc @leonardBang @ruanhang1993
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org