You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Prasanna kumar <pr...@gmail.com> on 2020/07/17 19:55:05 UTC
Flink Sinks
Hi ,
I did not find out of box flink sink connector for http and SQS mechanism.
Has anyone implemented it?
Wanted to know if we are writing a custom sink function , whether it
would affect semantic exactly one guarantees ?
Thanks ,
Prasanna
Re: Flink Sinks
Posted by David Anderson <da...@alpinegizmo.com>.
Prasanna,
The Flink project does not have an SQS connector, and a quick google search
hasn't found one. Nor does Flink have an HTTP sink, but with a bit of
googling you can find that various folks have implemented this themselves.
As for implementing SQS as a custom sink, if you need exactly once
guarantees I believe you will need to limit yourself to using SQS FIFO
queues, and you'll need to attach a MessageDeduplicationId to each record.
SQS offers to do deduplication for you, but only supports deduplication
within a 5 minute window -- which I would find too short to rely on during
recovery or redeployment.
See https://stackoverflow.com/a/28111986/2000823 for more on exactly once
with SQS.
Regards,
David
On Fri, Jul 17, 2020 at 9:55 PM Prasanna kumar <
prasannakumarramani@gmail.com> wrote:
> Hi ,
>
> I did not find out of box flink sink connector for http and SQS
> mechanism.
>
> Has anyone implemented it?
> Wanted to know if we are writing a custom sink function , whether it
> would affect semantic exactly one guarantees ?
>
>
> Thanks ,
> Prasanna
>
Re: Flink Sinks
Posted by David Anderson <da...@alpinegizmo.com>.
Prasanna,
The Flink project does not have an SQS connector, and a quick google search
hasn't found one. Nor does Flink have an HTTP sink, but with a bit of
googling you can find that various folks have implemented this themselves.
As for implementing SQS as a custom sink, if you need exactly once
guarantees I believe you will need to limit yourself to using SQS FIFO
queues, and you'll need to attach a MessageDeduplicationId to each record.
SQS offers to do deduplication for you, but only supports deduplication
within a 5 minute window -- which I would find too short to rely on during
recovery or redeployment.
See https://stackoverflow.com/a/28111986/2000823 for more on exactly once
with SQS.
Regards,
David
On Fri, Jul 17, 2020 at 9:55 PM Prasanna kumar <
prasannakumarramani@gmail.com> wrote:
> Hi ,
>
> I did not find out of box flink sink connector for http and SQS
> mechanism.
>
> Has anyone implemented it?
> Wanted to know if we are writing a custom sink function , whether it
> would affect semantic exactly one guarantees ?
>
>
> Thanks ,
> Prasanna
>