[jira] [Updated] (FLINK-24228) [FLIP-171] Firehose implementation of Async Sink

["Flink Jira Bot (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLINK-24228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Flink Jira Bot updated FLINK-24228:
-----------------------------------
    Labels: pull-request-available stale-assigned  (was: pull-request-available)

I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issue is assigned but has not received an update in 30 days, so it has been labeled "stale-assigned".
If you are still working on the issue, please remove the label and add a comment updating the community on your progress.  If this issue is waiting on feedback, please consider this a reminder to the committer/reviewer. Flink is a very active project, and so we appreciate your patience.
If you are no longer working on the issue, please unassign yourself so someone else may work on it.


> [FLIP-171] Firehose implementation of Async Sink
> ------------------------------------------------
>
>                 Key: FLINK-24228
>                 URL: https://issues.apache.org/jira/browse/FLINK-24228
>             Project: Flink
>          Issue Type: New Feature
>          Components: Connectors / Common
>            Reporter: Zichen Liu
>            Assignee: Zichen Liu
>            Priority: Major
>              Labels: pull-request-available, stale-assigned
>             Fix For: 1.15.0
>
>
> h2. Motivation
> *User stories:*
>  As a Flink user, I’d like to use Kinesis Firehose as sink for my data pipeline.
> *Scope:*
>  * Implement an asynchronous sink for Kinesis Firehose by inheriting the AsyncSinkBase class. The implementation can for now reside in its own module in flink-connectors. The module and package name can be anything reasonable e.g. {{flink-connector-aws-kinesis}} for the module name and {{org.apache.flink.connector.aws.kinesis}} for the package name.
>  * The implementation must use [the Kinesis Java Client|https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/kinesis/AmazonKinesisClient.html].
>  * The implementation must allow users to configure the Kinesis Client, with reasonable default settings.
>  * Implement an asynchornous sink writer for Firehose by extending the AsyncSinkWriter. The implementation must deal with failed requests and retry them using the {{requeueFailedRequestEntry}} method. If possible, the implementation should batch multiple requests (PutRecordsRequestEntry objects) to Firehose for increased throughput. The implemented Sink Writer will be used by the Sink class that will be created as part of this story.
>  * Unit/Integration testing. Use Kinesalite (in-memory Kinesis simulation). We already use this in {{KinesisTableApiITCase}}.
>  * Java / code-level docs.
>  * End to end testing: add tests that hits a real AWS instance. (How to best donate resources to the Flink project to allow this to happen?)
> h2. References
> More details to be found [https://cwiki.apache.org/confluence/display/FLINK/FLIP-171%3A+Async+Sink]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)