Observation regarding PutKafka while implementing NIFI-1672

[Pierre Villard <pi...@gmail.com>]

Hi,

I was working on NIFI-1672 [1], and while performing some tests with a
local running Kafka instance, I noticed there is a possible error regarding
how the behavior of the processor is documented.

The "Message delimiter" property is documented with:

"Specifies the delimiter (interpreted in its UTF-8 byte representation) to
use for splitting apart multiple messages within a single FlowFile. If not
specified, the entire content of the FlowFile will be used as a single
message. If specified, the contents of the FlowFile will be split on this
delimiter and each section sent as a separate Kafka message. Note that if
messages are delimited and some messages for a given FlowFile are
transferred successfully while others are not, the messages will be split
into individual FlowFiles, such that those messages that were successfully
sent are routed to the 'success' relationship while other messages are sent
to the 'failure' relationship."

I believe that the part "Note that if messages are delimited and some
messages for a given FlowFile are transferred successfully while others are
not, the messages will be split into individual FlowFiles, such that those
messages that were successfully sent are routed to the 'success'
relationship while other messages are sent to the 'failure' relationship."
is incorrect.

Instead I would say that the behavior (at least, it is what I observed) is:
if one or multiple messages inside the FlowFile are not transferred
successfully to Kafka, those messages are "tagged" thanks to some custom
attributes of the FlowFile and the whole FlowFile is sent to 'failure'
relationship. In case the 'failure' relationship is plugged back on the
PutKafka processor, only the messages previously in error will be sent to
Kafka.

I believe the documentation should be updated to reflect the current
behavior. Should I do it as part of NIFI-1672? Or create a specific JIRA
for that? In addition, should I create a JIRA to implement the behavior
where FlowFile are splitted as currently documented?

Since Kafka processors have been source of discussions lately, advice and
comments are welcomed!

Thanks !
Pierre

[1] https://issues.apache.org/jira/browse/NIFI-1672

Re: Observation regarding PutKafka while implementing NIFI-1672

[Oleg Zhurakousky <oz...@hortonworks.com>]

Pierre

You're absolutely correct. The explanation was valid, but there were problems with the original implementation of how it was described, so that changed, but we didn't update the docs


> On Apr 13, 2016, at 16:24, Pierre Villard <pi...@gmail.com> wrote:
> 
> Hi,
> 
> I was working on NIFI-1672 [1], and while performing some tests with a
> local running Kafka instance, I noticed there is a possible error regarding
> how the behavior of the processor is documented.
> 
> The "Message delimiter" property is documented with:
> 
> "Specifies the delimiter (interpreted in its UTF-8 byte representation) to
> use for splitting apart multiple messages within a single FlowFile. If not
> specified, the entire content of the FlowFile will be used as a single
> message. If specified, the contents of the FlowFile will be split on this
> delimiter and each section sent as a separate Kafka message. Note that if
> messages are delimited and some messages for a given FlowFile are
> transferred successfully while others are not, the messages will be split
> into individual FlowFiles, such that those messages that were successfully
> sent are routed to the 'success' relationship while other messages are sent
> to the 'failure' relationship."
> 
> I believe that the part "Note that if messages are delimited and some
> messages for a given FlowFile are transferred successfully while others are
> not, the messages will be split into individual FlowFiles, such that those
> messages that were successfully sent are routed to the 'success'
> relationship while other messages are sent to the 'failure' relationship."
> is incorrect.
> 
> Instead I would say that the behavior (at least, it is what I observed) is:
> if one or multiple messages inside the FlowFile are not transferred
> successfully to Kafka, those messages are "tagged" thanks to some custom
> attributes of the FlowFile and the whole FlowFile is sent to 'failure'
> relationship. In case the 'failure' relationship is plugged back on the
> PutKafka processor, only the messages previously in error will be sent to
> Kafka.
> 
> I believe the documentation should be updated to reflect the current
> behavior. Should I do it as part of NIFI-1672? Or create a specific JIRA
> for that? In addition, should I create a JIRA to implement the behavior
> where FlowFile are splitted as currently documented?
> 
> Since Kafka processors have been source of discussions lately, advice and
> comments are welcomed!
> 
> Thanks !
> Pierre
> 
> [1] https://issues.apache.org/jira/browse/NIFI-1672