You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Chesnay Schepler <ch...@apache.org> on 2022/07/07 11:35:01 UTC
[DISCUSS] FLIP-251: Support collecting arbitrary number of streams
Hello,
I have created a FLIP for slightly extending the collect() API to
support collecting an arbitrary number of streams, to make it more
useful for testing complex workflows.
https://cwiki.apache.org/confluence/display/FLINK/FLIP-251%3A+Support+collecting+arbitrary+number+of+streams
Looking forward to any feedback.
Regards,
Chesnay
Re: [DISCUSS] FLIP-251: Support collecting arbitrary number of streams
Posted by Martijn Visser <ma...@apache.org>.
+1
Op zo 10 jul. 2022 om 20:30 schreef Konstantin Knauf <kn...@apache.org>:
> Makes sense to me. Thank you, Chesnay. +1
>
> David Anderson <da...@apache.org> schrieb am Fr., 8. Juli 2022, 13:56:
>
> > I've found that with our current tooling it's frustrating to try to write
> > good end-to-end tests for real-world jobs with multiple sinks.
> > DataStream#executeAndCollect() is okay for simple pipelines with one
> sink,
> > but in my opinion we do need something like FLIP-251.
> >
> > The proposed interface looks good to me. I look forward to trying it.
> >
> > Chesnay, thanks for putting this together!
> >
> > David
> >
> > On Thu, Jul 7, 2022 at 7:35 AM Chesnay Schepler <ch...@apache.org>
> > wrote:
> >
> > > Hello,
> > >
> > > I have created a FLIP for slightly extending the collect() API to
> > > support collecting an arbitrary number of streams, to make it more
> > > useful for testing complex workflows.
> > >
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-251%3A+Support+collecting+arbitrary+number+of+streams
> > >
> > > Looking forward to any feedback.
> > >
> > > Regards,
> > >
> > > Chesnay
> > >
> > >
> >
>
Re: [DISCUSS] FLIP-251: Support collecting arbitrary number of streams
Posted by Konstantin Knauf <kn...@apache.org>.
Makes sense to me. Thank you, Chesnay. +1
David Anderson <da...@apache.org> schrieb am Fr., 8. Juli 2022, 13:56:
> I've found that with our current tooling it's frustrating to try to write
> good end-to-end tests for real-world jobs with multiple sinks.
> DataStream#executeAndCollect() is okay for simple pipelines with one sink,
> but in my opinion we do need something like FLIP-251.
>
> The proposed interface looks good to me. I look forward to trying it.
>
> Chesnay, thanks for putting this together!
>
> David
>
> On Thu, Jul 7, 2022 at 7:35 AM Chesnay Schepler <ch...@apache.org>
> wrote:
>
> > Hello,
> >
> > I have created a FLIP for slightly extending the collect() API to
> > support collecting an arbitrary number of streams, to make it more
> > useful for testing complex workflows.
> >
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-251%3A+Support+collecting+arbitrary+number+of+streams
> >
> > Looking forward to any feedback.
> >
> > Regards,
> >
> > Chesnay
> >
> >
>
Re: [DISCUSS] FLIP-251: Support collecting arbitrary number of streams
Posted by David Anderson <da...@apache.org>.
I've found that with our current tooling it's frustrating to try to write
good end-to-end tests for real-world jobs with multiple sinks.
DataStream#executeAndCollect() is okay for simple pipelines with one sink,
but in my opinion we do need something like FLIP-251.
The proposed interface looks good to me. I look forward to trying it.
Chesnay, thanks for putting this together!
David
On Thu, Jul 7, 2022 at 7:35 AM Chesnay Schepler <ch...@apache.org> wrote:
> Hello,
>
> I have created a FLIP for slightly extending the collect() API to
> support collecting an arbitrary number of streams, to make it more
> useful for testing complex workflows.
>
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-251%3A+Support+collecting+arbitrary+number+of+streams
>
> Looking forward to any feedback.
>
> Regards,
>
> Chesnay
>
>