You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "pawel-big-lebowski (via GitHub)" <gi...@apache.org> on 2025/04/16 12:23:40 UTC
[PR] [LINEAGE] Support dataset type facet for Avro [flink-connector-kafka]
pawel-big-lebowski opened a new pull request, #171:
URL: https://github.com/apache/flink-connector-kafka/pull/171
With an implementation of lineage interfaces, Kafka source and sink provide `TypeDatasetFacet` which contains type information of data processed. This was working with extracting type information from `SerializationSchema`:
```
TypeExtractor.getParameterType(SerializationSchema.class, valueSerialization.getClass(), 0)
```
However, this ain't working for `ConfluentRegistryAvroSerializationSchema` due to type erasure.
As a workaround, `TypeDatasetFacet` can be extended to contain `SerializationSchema` when direct extraction of `TypeInformation` is not possible. This allows handling this case on the lineage listener side, while keeping flink-connector-kafka unaware of Avro type extraction logic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34466][LINEAGE] Support dataset type facet for Avro [flink-connector-kafka]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #171:
URL: https://github.com/apache/flink-connector-kafka/pull/171#discussion_r2096097472
##########
flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/lineage/TypeDatasetFacet.java:
##########
@@ -1,11 +1,26 @@
package org.apache.flink.connector.kafka.lineage;
import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.api.common.serialization.SerializationSchema;
import org.apache.flink.api.common.typeinfo.TypeInformation;
import org.apache.flink.streaming.api.lineage.LineageDatasetFacet;
+import java.util.Optional;
+
/** Facet definition to contain type information of source and sink. */
@PublicEvolving
public interface TypeDatasetFacet extends LineageDatasetFacet {
TypeInformation getTypeInformation();
+
+ /**
+ * Sometimes a sink implementing {@link TypeDatasetFacetProvider} is not able to extract type.
+ * This is happening for AvroSerializationSchema due to type erasure problem. In this case, it
+ * makes sense to expose SerializationSchema to the lineage consumer so that it can use it to
+ * extract type information.
+ *
+ * @return
+ */
+ default Optional<SerializationSchema> getSerializationSchema() {
Review Comment:
Shouldn't this interface live in Flink? And the logic to fallback to the serialization schema also be in Flink.
As I assume a File connector using the Avro format could hit this issue as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34466][LINEAGE] Support dataset type facet for Avro [flink-connector-kafka]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #171:
URL: https://github.com/apache/flink-connector-kafka/pull/171#issuecomment-4377165506
This PR has been closed since it has not had any activity in 120 days.
If you feel like this was a mistake, or you would like to continue working on it,
please feel free to re-open the PR and ask for a review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34466][LINEAGE] Support dataset type facet for Avro [flink-connector-kafka]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #171:
URL: https://github.com/apache/flink-connector-kafka/pull/171#issuecomment-4188404445
This PR is being marked as stale since it has not had any activity in the last 90 days.
If you would like to keep this PR alive, please leave a comment asking for a review.
If the PR has merge conflicts, update it with the latest from the base branch.
If you are having difficulty finding a reviewer, please reach out to the
community, contact details can be found here: https://flink.apache.org/what-is-flink/community/
If this PR is no longer valid or desired, please feel free to close it.
If no activity occurs in the next 30 days, it will be automatically closed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34466][LINEAGE] Support dataset type facet for Avro [flink-connector-kafka]
Posted by "Efrat19 (via GitHub)" <gi...@apache.org>.
Efrat19 commented on code in PR #171:
URL: https://github.com/apache/flink-connector-kafka/pull/171#discussion_r2659457226
##########
flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/sink/KafkaRecordSerializationSchemaBuilder.java:
##########
@@ -432,13 +433,17 @@ public Optional<TypeDatasetFacet> getTypeDatasetFacet() {
((ResultTypeQueryable<?>) this.valueSerializationSchema)
.getProducedType()));
} else {
- // gets type information from serialize method signature
- Type type =
- TypeExtractor.getParameterType(
- SerializationSchema.class, valueSerializationSchema.getClass(), 0);
try {
+ Type type =
+ TypeExtractor.getParameterType(
+ SerializationSchema.class,
+ valueSerializationSchema.getClass(),
+ 0);
+
return Optional.of(
new DefaultTypeDatasetFacet(TypeExtractor.createTypeInfo(type)));
+ } catch (InvalidTypesException e) {
Review Comment:
Wouldn't a type check beforehand be less expensive resource-wise? (Depending on how often do you expect to hit this fallback)
##########
flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/table/DynamicKafkaRecordSerializationSchema.java:
##########
@@ -211,12 +212,14 @@ public Optional<TypeDatasetFacet> getTypeDatasetFacet() {
new DefaultTypeDatasetFacet(
((ResultTypeQueryable<?>) this.valueSerialization).getProducedType()));
} else {
- // gets type information from serialize method signature
- Type type =
- TypeExtractor.getParameterType(
- SerializationSchema.class, valueSerialization.getClass(), 0);
try {
+ Type type =
+ TypeExtractor.getParameterType(
+ SerializationSchema.class, valueSerialization.getClass(), 0);
+
return Optional.of(new DefaultTypeDatasetFacet(TypeExtractor.createTypeInfo(type)));
+ } catch (InvalidTypesException e) {
Review Comment:
Same as https://github.com/apache/flink-connector-kafka/pull/171/changes#r2659457226
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34466][LINEAGE] Support dataset type facet for Avro [flink-connector-kafka]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #171:
URL: https://github.com/apache/flink-connector-kafka/pull/171#discussion_r2096097472
##########
flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/lineage/TypeDatasetFacet.java:
##########
@@ -1,11 +1,26 @@
package org.apache.flink.connector.kafka.lineage;
import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.api.common.serialization.SerializationSchema;
import org.apache.flink.api.common.typeinfo.TypeInformation;
import org.apache.flink.streaming.api.lineage.LineageDatasetFacet;
+import java.util.Optional;
+
/** Facet definition to contain type information of source and sink. */
@PublicEvolving
public interface TypeDatasetFacet extends LineageDatasetFacet {
TypeInformation getTypeInformation();
+
+ /**
+ * Sometimes a sink implementing {@link TypeDatasetFacetProvider} is not able to extract type.
+ * This is happening for AvroSerializationSchema due to type erasure problem. In this case, it
+ * makes sense to expose SerializationSchema to the lineage consumer so that it can use it to
+ * extract type information.
+ *
+ * @return
+ */
+ default Optional<SerializationSchema> getSerializationSchema() {
Review Comment:
Shouldn't this interface live in Flink? As I assume a File connector using the Avro format could hit this issue as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34466][LINEAGE] Support dataset type facet for Avro [flink-connector-kafka]
Posted by "ferenc-csaky (via GitHub)" <gi...@apache.org>.
ferenc-csaky commented on code in PR #171:
URL: https://github.com/apache/flink-connector-kafka/pull/171#discussion_r2535366840
##########
flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/lineage/TypeDatasetFacet.java:
##########
@@ -1,11 +1,26 @@
package org.apache.flink.connector.kafka.lineage;
import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.api.common.serialization.SerializationSchema;
import org.apache.flink.api.common.typeinfo.TypeInformation;
import org.apache.flink.streaming.api.lineage.LineageDatasetFacet;
+import java.util.Optional;
+
/** Facet definition to contain type information of source and sink. */
@PublicEvolving
public interface TypeDatasetFacet extends LineageDatasetFacet {
TypeInformation getTypeInformation();
+
+ /**
+ * Sometimes a sink implementing {@link TypeDatasetFacetProvider} is not able to extract type.
+ * This is happening for AvroSerializationSchema due to type erasure problem. In this case, it
+ * makes sense to expose SerializationSchema to the lineage consumer so that it can use it to
+ * extract type information.
+ *
+ * @return
+ */
+ default Optional<SerializationSchema> getSerializationSchema() {
Review Comment:
I strongly agree. The whole `TypeDatasetFacet` should be removed from here. As far as i can tell, this PR should be revived and merged beforehand: https://github.com/apache/flink/pull/25712
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34466][LINEAGE] Support dataset type facet for Avro [flink-connector-kafka]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #171: [FLINK-34466][LINEAGE] Support dataset type facet for Avro
URL: https://github.com/apache/flink-connector-kafka/pull/171
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org