[jira] [Created] (FLINK-26056) CLONE - [FLIP-171] KDS implementation of Async Sink Table API

["Ahmed Hamdy (Jira)" <ji...@apache.org>]

Ahmed Hamdy created FLINK-26056:
-----------------------------------

             Summary: CLONE - [FLIP-171] KDS implementation of Async Sink Table API
                 Key: FLINK-26056
                 URL: https://issues.apache.org/jira/browse/FLINK-26056
             Project: Flink
          Issue Type: New Feature
          Components: Connectors / Kinesis
            Reporter: Ahmed Hamdy
            Assignee: Ahmed Hamdy
             Fix For: 1.15.0


h2. Motivation

*User stories:*
As a Flink user, I’d like to use the Table API for the new Kinesis Data Streams  sink.

*Scope:*
 * Introduce {{AsyncDynamicTableSink}} that enables Sinking Tables into Async Implementations.
 * Implement a new {{KinesisDynamicTableSink}} that uses {{KinesisDataStreamSink}} Async Implementation and implements {{{}AsyncDynamicTableSink{}}}.
 * The implementation introduces Async Sink configurations as optional options in the table definition, with default values derived from the {{KinesisDataStream}} default values.
 * Unit/Integration testing. modify KinesisTableAPI tests for the new implementation, add unit tests for {{AsyncDynamicTableSink}} and {{KinesisDynamicTableSink}} and {{{}KinesisDynamicTableSinkFactory{}}}.
 * Java / code-level docs.

h2. References

More details to be found [https://cwiki.apache.org/confluence/display/FLINK/FLIP-171%3A+Async+Sink]

 

*Update:*

^^ Status Update ^^
__List of all work outstanding for 1.15 release__

[Merged] https://github.com/apache/flink/pull/18165 - KDS DataStream Docs
[Merged] https://github.com/apache/flink/pull/18396 - [hotfix] for infinte loop if not flushing during commit
[Merged] https://github.com/apache/flink/pull/18421 - Mark Kinesis Producer as deprecated (Prod: FLINK-24227)
[Merged] https://github.com/apache/flink/pull/18348 - KDS Table API Sink & Docs
[Merged] https://github.com/apache/flink/pull/18488 - base sink retry entries in order not in reverse
[Merged] https://github.com/apache/flink/pull/18512 - changing failed requests handler to accept List in AsyncSinkWriter
[Merged] https://github.com/apache/flink/pull/18483 - Do not expose the element converter
[Merged] https://github.com/apache/flink/pull/18468 - Adding Kinesis data streams sql uber-jar

Ready for review:
[SUCCESS ] https://github.com/apache/flink/pull/18314 - KDF DataStream Sink & Docs
[BLOCKED on ^^ ] https://github.com/apache/flink/pull/18426 - rename flink-connector-aws-kinesis-data-* to flink-connector-aws-kinesis-* (module names) and KinesisData*Sink to Kinesis*Sink (class names)

Pending PR:
* Firehose Table API Sink & Docs
* KDF Table API SQL jar

TBD:
* FLINK-25846: Not shutting down
* FLINK-25848: Validation during start up
* FLINK-25792: flush() bug
* FLINK-25793: throughput exceeded
* Update the defaults of KDS sink and update the docs + do the same for KDF
* add a `AsyncSinkCommonConfig` class (to wrap the 6 params) to the `flink-connector-base` and propagate it to the two connectors
------------- feature freeze
* KDS performance testing
* KDF performance testing
* Clone the new docs to .../contents.zh/... and add the location to the corresponding Chinese translation jira - KDS -
* Rename [Amazon AWS Kinesis Streams] to [Amazon Kinesis Data Streams] in docs (legacy issue)
------------- Flink 1.15 release
* KDS end to end sanity test - hits aws apis rather than local docker images
* KDS Python wrappers
* FLINK-25733 - Create A migration guide for Kinesis Table API connector - can happen after 1.15
* If `endpoint` is provided, `region` should not be required like it currently is
* Test if Localstack container requires the 10000ms timeout
* Adaptive level of logging (in discussion)

FYI:
* FLINK-25661 - Add Custom Fatal Exception handler in AsyncSinkWriter - https://github.com/apache/flink/pull/18449
* https://issues.apache.org/jira/browse/FLINK-24229 DDB Sink

Chinese translation:
https://issues.apache.org/jira/browse/FLINK-25735 - KDS DataStream Sink
https://issues.apache.org/jira/browse/FLINK-25736 - KDS Table API Sink
https://issues.apache.org/jira/browse/FLINK-25737 - KDF DataStream Sink



--
This message was sent by Atlassian Jira
(v8.20.1#820001)