You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Xiaolong Wang <xi...@smartnews.com> on 2020/05/13 05:43:56 UTC
How To subscribe a Kinesis Stream using enhance fanout?
Hello Flink Community!
I'm currently coding on a project relying on AWS Kinesis. With the
provided connector (flink-connector-kinesis_2.11;1.10.0), I can consume the
message.
But as the main stream is used among several other teams, I was
required to use the enhance fanout of Kinesis. I checked the connector code
and found no implementations.
Has this issue occurred to anyone before?
Thanks for your help.
Re: How To subscribe a Kinesis Stream using enhance fanout?
Posted by "Tzu-Li (Gordon) Tai" <tz...@apache.org>.
Hi all,
Just to give a quick update: there will be contributors from the AWS
Kinesis team working on contributing enhanced fan out support to the
connector.
You can follow the progress here:
https://issues.apache.org/jira/browse/FLINK-17688
Cheers,
Gordon
On Fri, May 15, 2020 at 5:55 PM orionemail <or...@protonmail.com>
wrote:
> Hi,
>
> We also recently needed this functionality, unfortunately we were unable
> to implement it ourselves so changed our plan accordingly.
>
> However we very much see the benefit for this feature and would be
> interested in following the JIRA ticket.
>
> Thanks
>
>
>
> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> On Thursday, 14 May 2020 11:34, Xiaolong Wang <xi...@smartnews.com>
> wrote:
>
> Thanks, I'll check it out.
>
> On Thu, May 14, 2020 at 6:26 PM Tzu-Li (Gordon) Tai <tz...@apache.org>
> wrote:
>
>> Hi Xiaolong,
>>
>> You are right, the way the Kinesis connector is implemented / the way the
>> AWS APIs are used, does not allow it to consume Kinesis streams with
>> enhanced fan-out enabled consumers [1].
>> Could you open a JIRA ticket for this?
>> As far as I can tell, this could be a valuable contribution to the
>> connector for Kinesis users who require dedicated throughput isolated from
>> other running consumers.
>>
>> Cheers,
>> Gordon
>>
>> [1]
>> https://docs.aws.amazon.com/streams/latest/dev/building-enhanced-consumers-api.html
>>
>> On Wed, May 13, 2020 at 1:44 PM Xiaolong Wang <
>> xiaolong.wang@smartnews.com> wrote:
>>
>>> Hello Flink Community!
>>>
>>> I'm currently coding on a project relying on AWS Kinesis. With the
>>> provided connector (flink-connector-kinesis_2.11;1.10.0), I can consume the
>>> message.
>>>
>>> But as the main stream is used among several other teams, I was
>>> required to use the enhance fanout of Kinesis. I checked the connector code
>>> and found no implementations.
>>>
>>> Has this issue occurred to anyone before?
>>>
>>> Thanks for your help.
>>>
>>
>
Re: How To subscribe a Kinesis Stream using enhance fanout?
Posted by orionemail <or...@protonmail.com>.
Hi,
We also recently needed this functionality, unfortunately we were unable to implement it ourselves so changed our plan accordingly.
However we very much see the benefit for this feature and would be interested in following the JIRA ticket.
Thanks
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Thursday, 14 May 2020 11:34, Xiaolong Wang <xi...@smartnews.com> wrote:
> Thanks, I'll check it out.
>
> On Thu, May 14, 2020 at 6:26 PM Tzu-Li (Gordon) Tai <tz...@apache.org> wrote:
>
>> Hi Xiaolong,
>>
>> You are right, the way the Kinesis connector is implemented / the way the AWS APIs are used, does not allow it to consume Kinesis streams with enhanced fan-out enabled consumers [1].
>> Could you open a JIRA ticket for this?
>> As far as I can tell, this could be a valuable contribution to the connector for Kinesis users who require dedicated throughput isolated from other running consumers.
>>
>> Cheers,
>> Gordon
>>
>> [1] https://docs.aws.amazon.com/streams/latest/dev/building-enhanced-consumers-api.html
>>
>> On Wed, May 13, 2020 at 1:44 PM Xiaolong Wang <xi...@smartnews.com> wrote:
>>
>>> Hello Flink Community!
>>>
>>> I'm currently coding on a project relying on AWS Kinesis. With the provided connector (flink-connector-kinesis_2.11;1.10.0), I can consume the message.
>>>
>>> But as the main stream is used among several other teams, I was required to use the enhance fanout of Kinesis. I checked the connector code and found no implementations.
>>>
>>> Has this issue occurred to anyone before?
>>>
>>> Thanks for your help.
Re: How To subscribe a Kinesis Stream using enhance fanout?
Posted by Xiaolong Wang <xi...@smartnews.com>.
Thanks, I'll check it out.
On Thu, May 14, 2020 at 6:26 PM Tzu-Li (Gordon) Tai <tz...@apache.org>
wrote:
> Hi Xiaolong,
>
> You are right, the way the Kinesis connector is implemented / the way the
> AWS APIs are used, does not allow it to consume Kinesis streams with
> enhanced fan-out enabled consumers [1].
> Could you open a JIRA ticket for this?
> As far as I can tell, this could be a valuable contribution to the
> connector for Kinesis users who require dedicated throughput isolated from
> other running consumers.
>
> Cheers,
> Gordon
>
> [1]
> https://docs.aws.amazon.com/streams/latest/dev/building-enhanced-consumers-api.html
>
> On Wed, May 13, 2020 at 1:44 PM Xiaolong Wang <xi...@smartnews.com>
> wrote:
>
>> Hello Flink Community!
>>
>> I'm currently coding on a project relying on AWS Kinesis. With the
>> provided connector (flink-connector-kinesis_2.11;1.10.0), I can consume the
>> message.
>>
>> But as the main stream is used among several other teams, I was
>> required to use the enhance fanout of Kinesis. I checked the connector code
>> and found no implementations.
>>
>> Has this issue occurred to anyone before?
>>
>> Thanks for your help.
>>
>
Re: How To subscribe a Kinesis Stream using enhance fanout?
Posted by "Tzu-Li (Gordon) Tai" <tz...@apache.org>.
Hi Xiaolong,
You are right, the way the Kinesis connector is implemented / the way the
AWS APIs are used, does not allow it to consume Kinesis streams with
enhanced fan-out enabled consumers [1].
Could you open a JIRA ticket for this?
As far as I can tell, this could be a valuable contribution to the
connector for Kinesis users who require dedicated throughput isolated from
other running consumers.
Cheers,
Gordon
[1]
https://docs.aws.amazon.com/streams/latest/dev/building-enhanced-consumers-api.html
On Wed, May 13, 2020 at 1:44 PM Xiaolong Wang <xi...@smartnews.com>
wrote:
> Hello Flink Community!
>
> I'm currently coding on a project relying on AWS Kinesis. With the
> provided connector (flink-connector-kinesis_2.11;1.10.0), I can consume the
> message.
>
> But as the main stream is used among several other teams, I was
> required to use the enhance fanout of Kinesis. I checked the connector code
> and found no implementations.
>
> Has this issue occurred to anyone before?
>
> Thanks for your help.
>