You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@nifi.apache.org by "Josh Highley (Jira)" <ji...@apache.org> on 2021/09/27 18:07:00 UTC

[jira] [Created] (NIFI-9250) PublishKafka, PublishKafkaRecord: send tombstone messages

Josh Highley created NIFI-9250:
----------------------------------

             Summary: PublishKafka, PublishKafkaRecord: send tombstone messages
                 Key: NIFI-9250
                 URL: https://issues.apache.org/jira/browse/NIFI-9250
             Project: Apache NiFi
          Issue Type: Improvement
            Reporter: Josh Highley


The Kafka Connect JDBC Sink Connector supports database deletes using what's called tombstone messages.  The Kafka message key contains the primary key of the database record to be deleted and the message value is NULL.

The Publish Kafka processors do not appear to support the necessary formats for the key or value: 

1) There's not a way to send a null message value. 

2) If the DB key is a single column, the message key can just be the value. However, if the primary key is a composite key (more than 1 column) then the message key needs a schema. This could be JSON with embedded schema but this is inefficient. Ideally, I'd be able to write an Avro message with a schema registry reference to the Kafka message key.

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)