[jira] [Commented] (KAFKA-8314) Managing the doc field in case of schema projection - kafka connect

["kaushik srinivas (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-8314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16831549#comment-16831549 ] 

kaushik srinivas commented on KAFKA-8314:
-----------------------------------------

We have added a check in connect/api/src/main/java/org/apache/kafka/connect/data/SchemaProjector.java

as below to exclude non mandatory items of schema in our local build of kafka.
{code:java}
private static void checkMaybeCompatible(Schema source, Schema target) {
if (source.type() != target.type() && !isPromotable(source.type(), target.type())) {
throw new SchemaProjectorException("Schema type mismatch. source type: " + source.type() + " and target type: " + target.type());
} else if (!Objects.equals(source.name(), target.name())) {
throw new SchemaProjectorException("Schema name mismatch. source name: " + source.name() + " and target name: " + target.name());
} else if (!Objects.equals(source.parameters(), target.parameters())) {

// Create a copy of the original source and schema properties maps.
Map<String,String> sourceParameters = new HashMap<>(source.parameters());
Map<String,String> targetParameters = new HashMap<>(target.parameters());

// exclude doc,aliases and namespace fields for schema compatability check
sourceParameters.remove("connect.record.doc");
sourceParameters.remove("connect.record.aliases");
sourceParameters.remove("connect.record.namespace");
targetParameters.remove("connect.record.doc");
targetParameters.remove("connect.record.aliases");
targetParameters.remove("connect.record.namespace");

// throw SchemaProjectorException if comparison fails for remaining attributes.
if(!Objects.equals(sourceParameters, targetParameters)) {
throw new SchemaProjectorException("Schema parameters not equal. source parameters: " + source.parameters() + " and target parameters: " + target.parameters());
}
}
{code}
 

> Managing the doc field in case of schema projection - kafka connect
> -------------------------------------------------------------------
>
>                 Key: KAFKA-8314
>                 URL: https://issues.apache.org/jira/browse/KAFKA-8314
>             Project: Kafka
>          Issue Type: Bug
>            Reporter: kaushik srinivas
>            Priority: Major
>
> Doc field change in the schema while writing to hdfs using hdfs sink connector via connect framework would cause failures in schema projection.
>  
> java.lang.RuntimeException: org.apache.kafka.connect.errors.SchemaProjectorException: Schema parameters not equal. source parameters: \{connect.record.doc=xxx} and target parameters: \{connect.record.doc=yyy} 
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)