[jira] [Comment Edited] (FLINK-24229) [FLIP-171] DynamoDB implementation of Async Sink

["Yuri Gusev (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-24229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17504922#comment-17504922 ] 

Yuri Gusev edited comment on FLINK-24229 at 3/11/22, 1:34 PM:
--------------------------------------------------------------

Ok thank you [~martijnvisser] and [~CrynetLogistics] .

There were some changes on the master and no there are conflicts so I'll rebase our job on top of the latest changes.

[~CrynetLogistics]  would you mind taking a look at the question we had in the PR about elementConverter being hidden? We have a sample PR ([see Nir's comment|https://github.com/apache/flink/pull/18518#discussion_r799306670]) on what it would take to hide elementConverter. But that might impact performance of the sink quite a lot.

We will wait with the documentation ticket until we agree with the reviewers on the ElementConverter changes. 


was (Author: gusev):
Ok thank you [~martijnvisser] and [~CrynetLogistics] .

There were some changes on the master and no there are conflicts so I'll rebase our job on top of the latest changes.

[~CrynetLogistics]  would you mind taking a look at the question we had in the PR about elementConverter being hidden. He have a sample PR ([see Nir's comment|https://github.com/apache/flink/pull/18518#discussion_r799306670]) on what it would take to hide elementConverter. But that might impact performance of the sink quite a lot.

We will wait with the documentation ticket until we agree with the reviewers on the ElementConverter changes. 

> [FLIP-171] DynamoDB implementation of Async Sink
> ------------------------------------------------
>
>                 Key: FLINK-24229
>                 URL: https://issues.apache.org/jira/browse/FLINK-24229
>             Project: Flink
>          Issue Type: New Feature
>          Components: Connectors / Common
>            Reporter: Zichen Liu
>            Assignee: Yuri Gusev
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 1.16.0
>
>
> h2. Motivation
> *User stories:*
>  As a Flink user, I’d like to use DynamoDB as sink for my data pipeline.
> *Scope:*
>  * Implement an asynchronous sink for DynamoDB by inheriting the AsyncSinkBase class. The implementation can for now reside in its own module in flink-connectors.
>  * Implement an asynchornous sink writer for DynamoDB by extending the AsyncSinkWriter. The implementation must deal with failed requests and retry them using the {{requeueFailedRequestEntry}} method. If possible, the implementation should batch multiple requests (PutRecordsRequestEntry objects) to Firehose for increased throughput. The implemented Sink Writer will be used by the Sink class that will be created as part of this story.
>  * Java / code-level docs.
>  * End to end testing: add tests that hits a real AWS instance. (How to best donate resources to the Flink project to allow this to happen?)
> h2. References
> More details to be found [https://cwiki.apache.org/confluence/display/FLINK/FLIP-171%3A+Async+Sink]



--
This message was sent by Atlassian Jira
(v8.20.1#820001)