Kafka record producer + consumer issues

[Michael Karpel <pl...@gmail.com>]

Hello
We are using Nifi 1.9 with [Consume/Publish]KafkaRecord_2_0 processors and
Confluent Content-Encoded Schema Reference...
All works fine regarding the payload, but we also use Key (which is also
encoded in the same way)
As expected if we use different schema or different kafka headers the
messages are not placed in the same flow file (as documented and expected),
however there is no guarantees for the key, in ConsumeKafka_2_0 the key is
part of the flow file attributes and same key messages batch in the same
flow file...

I want to make a fork of the [Consume/Publish]KafkaRecord_2_0 and implement
the following spec, and want to ask if its consistent with the planned
future approach...

ConsumeKafkaRecordWithKey_2_0:
add property - Key Reader (RecordReader) optional
add property - Key Writer (RecordWriter) optional
parse the key with reader if set, write key with writer if set

FlowFileAttributeWriter (RecordWriter)
property - Record Type - Simple (for simple type keys) / Map
property - Flow File Attributes Prefix (for my case kafka.key)
will take logical record and write as map to flow file attributes

PublishKafkaRecordWithKey_2_0
add property - Key Reader (RecordReader) optional
add property - Key Writer (RecordWriter) optional
if Message Key Field set use as simple key
otherwise use Key Reader and Key Writer

FlowFileAttributeReader (recordReader)
property - Record Type - Simple (for simple type keys) / Map
property - Flow File Attributes regex (for my case kafka/.key/..*)
property - Flow File Attribute Level Demarcation (default to .)
will take flow file attributes and parse as logical record map

Is this approach consistent with the guidelines (reader/writer for flow
file attributes)?

BR,
Michael Karpel
Data engineering consultant | Matrix BI | +972 54 2680303 |
plexaikm@gmail.com