You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by "Zhijiang(wangzhijiang999)" <wa...@aliyun.com> on 2017/04/26 03:58:56 UTC
回复：Multiple consumers on a subpartition
Hi albert,
As I know, if the upstream data will be consumed by multiple consumers, it will generate multiple subpartitions, and each subpartition will correspond to one input channel consumer.So it is one-to-one correspondence among subpartition -> subpartition view -> input channel.
cheers,zhijiang------------------------------------------------------------------发件人：albertjonathan <al...@cs.umn.edu>发送时间：2017年4月26日(星期三) 02:37收件人：user <us...@flink.apache.org>主 题：Multiple consumers on a subpartition
Hello,
Is there a way Flink allow a (pipelined) subpartition to be consumed by
multiple consumers? If not, would it make more sense to implement it as
multiple input channels for a single subpartition or multiple subpartition
views for each input channel?
Any suggestion is appreciated.
Thanks
--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Multiple-consumers-on-a-subpartition-tp12809.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
Re: Multiple consumers on a subpartition
Posted by Albert Jonathan <al...@cs.umn.edu>.
Thank you for your responses and suggestions. I appreciate it.
Albert
On Wed, Apr 26, 2017 at 4:19 AM, Ufuk Celebi <uc...@apache.org> wrote:
> Adding to what Zhijiang said: I think the way to go would be to create
> multiple "read views" over the pipelined subpartition. You would have
> to make sure that the initial reference count of the partition buffers
> is incremented accordingly. The producer will be back pressured by
> both consumers now. This could be undesired in some scenarios.
> Currently, both consumers are independent of each other by creating
> multiple partitions (with their own subpartitions) for each consumer.
>
>
>
> On Wed, Apr 26, 2017 at 5:58 AM, Zhijiang(wangzhijiang999)
> <wa...@aliyun.com> wrote:
> > Hi albert,
> >
> > As I know, if the upstream data will be consumed by multiple
> consumers,
> > it will generate multiple subpartitions, and each subpartition will
> > correspond to one input channel consumer.
> > So it is one-to-one correspondence among subpartition -> subpartition
> view
> > -> input channel.
> >
> > cheers,
> > zhijiang
> >
> > ------------------------------------------------------------------
> > 发件人：albertjonathan <al...@cs.umn.edu>
> > 发送时间：2017年4月26日(星期三) 02:37
> > 收件人：user <us...@flink.apache.org>
> > 主 题：Multiple consumers on a subpartition
> >
> > Hello,
> >
> > Is there a way Flink allow a (pipelined) subpartition to be consumed by
> > multiple consumers? If not, would it make more sense to implement it as
> > multiple input channels for a single subpartition or multiple
> subpartition
> > views for each input channel?
> >
> > Any suggestion is appreciated.
> >
> > Thanks
> >
> >
> >
> > --
> > View this message in context:
> > http://apache-flink-user-mailing-list-archive.2336050.
> n4.nabble.com/Multiple-consumers-on-a-subpartition-tp12809.html
> > Sent from the Apache Flink User Mailing List archive. mailing list
> archive
> > at Nabble.com.
> >
> >
>
Re: Multiple consumers on a subpartition
Posted by Ufuk Celebi <uc...@apache.org>.
Adding to what Zhijiang said: I think the way to go would be to create
multiple "read views" over the pipelined subpartition. You would have
to make sure that the initial reference count of the partition buffers
is incremented accordingly. The producer will be back pressured by
both consumers now. This could be undesired in some scenarios.
Currently, both consumers are independent of each other by creating
multiple partitions (with their own subpartitions) for each consumer.
On Wed, Apr 26, 2017 at 5:58 AM, Zhijiang(wangzhijiang999)
<wa...@aliyun.com> wrote:
> Hi albert,
>
> As I know, if the upstream data will be consumed by multiple consumers,
> it will generate multiple subpartitions, and each subpartition will
> correspond to one input channel consumer.
> So it is one-to-one correspondence among subpartition -> subpartition view
> -> input channel.
>
> cheers,
> zhijiang
>
> ------------------------------------------------------------------
> 发件人：albertjonathan <al...@cs.umn.edu>
> 发送时间：2017年4月26日(星期三) 02:37
> 收件人：user <us...@flink.apache.org>
> 主 题：Multiple consumers on a subpartition
>
> Hello,
>
> Is there a way Flink allow a (pipelined) subpartition to be consumed by
> multiple consumers? If not, would it make more sense to implement it as
> multiple input channels for a single subpartition or multiple subpartition
> views for each input channel?
>
> Any suggestion is appreciated.
>
> Thanks
>
>
>
> --
> View this message in context:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Multiple-consumers-on-a-subpartition-tp12809.html
> Sent from the Apache Flink User Mailing List archive. mailing list archive
> at Nabble.com.
>
>