[jira] [Commented] (KAFKA-8314) Managing the doc field in case of schema projection - kafka connect

["Karim Lamouri (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-8314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17313442#comment-17313442 ] 

Karim Lamouri commented on KAFKA-8314:
--------------------------------------

We have the same problem on Kafka 2.5.1.

Changing the doc cause issues with Kafka-Connect

 
{code:java}
org.apache.kafka.connect.errors.ConnectException: Exiting WorkerSinkTask due to unrecoverable exception.
	at org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:568)
	at org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:326)
	at org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:229)
	at org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:201)
	at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:185)
	at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:235)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.kafka.connect.errors.ConnectException: org.apache.kafka.connect.errors.SchemaProjectorException: Error projecting auction
	at io.confluent.connect.s3.TopicPartitionWriter.write(TopicPartitionWriter.java:190)
	at io.confluent.connect.s3.S3SinkTask.put(S3SinkTask.java:191)
	at org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:546)
	... 10 more
Caused by: org.apache.kafka.connect.errors.SchemaProjectorException: Error projecting auction
	at org.apache.kafka.connect.data.SchemaProjector.projectStruct(SchemaProjector.java:113)
	at org.apache.kafka.connect.data.SchemaProjector.projectRequiredSchema(SchemaProjector.java:93)
	at org.apache.kafka.connect.data.SchemaProjector.project(SchemaProjector.java:73)
	at io.confluent.connect.storage.schema.StorageSchemaCompatibility.projectInternal(StorageSchemaCompatibility.java:395)
	at io.confluent.connect.storage.schema.StorageSchemaCompatibility.projectInternal(StorageSchemaCompatibility.java:383)
	at io.confluent.connect.storage.schema.StorageSchemaCompatibility.project(StorageSchemaCompatibility.java:355)
	at io.confluent.connect.s3.TopicPartitionWriter.checkRotationOrAppend(TopicPartitionWriter.java:269)
	at io.confluent.connect.s3.TopicPartitionWriter.executeState(TopicPartitionWriter.java:219)
	at io.confluent.connect.s3.TopicPartitionWriter.write(TopicPartitionWriter.java:188)
	... 12 more
Caused by: org.apache.kafka.connect.errors.SchemaProjectorException: Error projecting site
	at org.apache.kafka.connect.data.SchemaProjector.projectStruct(SchemaProjector.java:113)
	at org.apache.kafka.connect.data.SchemaProjector.projectRequiredSchema(SchemaProjector.java:93)
	at org.apache.kafka.connect.data.SchemaProjector.project(SchemaProjector.java:73)
	at org.apache.kafka.connect.data.SchemaProjector.projectStruct(SchemaProjector.java:110)
	... 20 more
Caused by: org.apache.kafka.connect.errors.SchemaProjectorException: Schema parameters not equal. source parameters: XXXX 
{code}
With the only difference between the source and the target parameters being the addition of:

{{io.confluent.connect.avro.field.doc.FieldName=new documentation for new field}} 

Only way to fix this situation is to have schema evolution set to NONE in Kafka-Connect

> Managing the doc field in case of schema projection - kafka connect
> -------------------------------------------------------------------
>
>                 Key: KAFKA-8314
>                 URL: https://issues.apache.org/jira/browse/KAFKA-8314
>             Project: Kafka
>          Issue Type: Bug
>            Reporter: kaushik srinivas
>            Priority: Major
>
> Doc field change in the schema while writing to hdfs using hdfs sink connector via connect framework would cause failures in schema projection.
>  
> java.lang.RuntimeException: org.apache.kafka.connect.errors.SchemaProjectorException: Schema parameters not equal. source parameters: \{connect.record.doc=xxx} and target parameters: \{connect.record.doc=yyy} 
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)