[jira] [Created] (FLINK-39813) [Connectors/Kinesis] Add lineage support to Kinesis Streams connector

["Francisco Morillo (Jira)" <ji...@apache.org>]

Francisco Morillo created FLINK-39813:
-----------------------------------------

             Summary: [Connectors/Kinesis] Add lineage support to Kinesis Streams connector
                 Key: FLINK-39813
                 URL: https://issues.apache.org/jira/browse/FLINK-39813
             Project: Flink
          Issue Type: Improvement
          Components: Connectors / Kinesis
    Affects Versions: aws-connector-6.0.0
            Reporter: Francisco Morillo


  The Kinesis Streams connector (flink-connector-aws-kinesis-streams) does not implement the LineageVertexProvider
  interface introduced in Flink 2.0. This means lineage tools like OpenLineage cannot automatically extract dataset
  information from Flink jobs using KDS sources and sinks.

  The Kafka connector already implements this successfully. This change adds the same capability to the Kinesis Streams
  connector.

  Implementation:
  - KinesisStreamsSource implements LineageVertexProvider, returning SourceLineageVertex with namespace derived from
  stream ARN
  - KinesisStreamsSink implements LineageVertexProvider, returning LineageVertex with the same ARN-based namespace
  - New utility package: KinesisLineageUtil, KinesisDatasetFacet, TypeDatasetFacet
  - Unit tests covering source, sink, and utility methods

  Namespace format: arn:\{partition}:kinesis:\{region}:\{account}:stream
  Dataset name: stream name extracted from the ARN

  This covers:
  - DataStream API (KinesisStreamsSource/Sink implement the interface directly)
  - SQL/Table API (KinesisDynamicSource internally creates KinesisStreamsSource, planner extracts lineage automatically)

  End-to-end tested with OpenLineage flink2 module (v1.47.1) confirming lineage events are emitted with correct
  namespace and stream name when using the DataStream API.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)