You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "Pierre Villard (Jira)" <ji...@apache.org> on 2023/11/15 10:50:00 UTC
[jira] [Updated] (NIFI-12371) Support tombstone messages in non-record Kafka processors

     [ https://issues.apache.org/jira/browse/NIFI-12371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Pierre Villard updated NIFI-12371:
----------------------------------
    Description: 
In some scenarios (compaction, transactions, etc), you may have Kafka tombstone messages (ie. the message has null for the content) and it may be important to have NiFi properly consume and publish those tombstone messages. As of now, when consuming Kafka messages, there is no way to differentiate a message that would be null (tombstone) and a message with an empty string). Also, as of now, when publishing a Kafka message, there is no way to send a null message, it would always be an empty string (empty byte array).

This JIRA is about adding support for tombstone messages in ConsumeKafka_2_6 and PublishKafka_2_6 when not using a message demarcator. Follow up JIRAs could be filed to add support in the record based processors.

  was:
In some scenarios (compaction, transactions, etc), you may have Kafka tombstone messages (ie. the message has null for the content) and it may be important to have NiFi properly consume and publish those tombstone messages. As of now, when consuming Kafka messages, there is no way to differentiate a message that would be null (tombstone) and a message with an empty string). Also, as of now, when publishing a Kafka message, there is no way to send a null message, it would always be an empty string (empty byte array).

This JIRA is about adding support for tombstone messages in ConsumeKafka_2_6 and PublishKafka_2_6 when not using a delimiter. Follow up JIRAs could be filed to add support in the record based processors.


> Support tombstone messages in non-record Kafka processors
> ---------------------------------------------------------
>
>                 Key: NIFI-12371
>                 URL: https://issues.apache.org/jira/browse/NIFI-12371
>             Project: Apache NiFi
>          Issue Type: Improvement
>          Components: Extensions
>            Reporter: Pierre Villard
>            Assignee: Pierre Villard
>            Priority: Minor
>
> In some scenarios (compaction, transactions, etc), you may have Kafka tombstone messages (ie. the message has null for the content) and it may be important to have NiFi properly consume and publish those tombstone messages. As of now, when consuming Kafka messages, there is no way to differentiate a message that would be null (tombstone) and a message with an empty string). Also, as of now, when publishing a Kafka message, there is no way to send a null message, it would always be an empty string (empty byte array).
> This JIRA is about adding support for tombstone messages in ConsumeKafka_2_6 and PublishKafka_2_6 when not using a message demarcator. Follow up JIRAs could be filed to add support in the record based processors.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)