[jira] [Updated] (NIFI-14394) ConsumeKinesisStream support for Sub Sequence Number

["David Handermann (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/NIFI-14394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

David Handermann updated NIFI-14394:
------------------------------------
    Affects Version/s:     (was: 2.3.0)

> ConsumeKinesisStream support for Sub Sequence Number
> ----------------------------------------------------
>
>                 Key: NIFI-14394
>                 URL: https://issues.apache.org/jira/browse/NIFI-14394
>             Project: Apache NiFi
>          Issue Type: Improvement
>          Components: Extensions
>            Reporter: Dariusz Seweryn
>            Priority: Major
>          Time Spent: 20m
>  Remaining Estimate: 0h
>
> ConsumeKinesisStream supports "Output Strategy" of "Use Wrapper" which wraps message data and metadata into record content.
> Currently metadata includes:
>  * stream
>  * shardId
>  * sequenceNumber
>  * partitionKey
>  * approximateArrival
> What it does not include is Sub Sequence Number which is assigned to messages that [are aggregated by the Amazon's Kinesis Producer Library|https://docs.aws.amazon.com/streams/latest/dev/kinesis-producer-adv-aggregation.html] making them indistinguishable by evaluating metadata.
> Proposal is to add to wrapped metadata:
>  * subSequenceNumber — a long
>  * shardUniqueId — a String formed as {{String.format("%s%020d", sequenceNumber, subSequenceNumber)}} which could be useful for uniquely identifying messages identifying order using a single comparison



--
This message was sent by Atlassian Jira
(v8.20.10#820010)