You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Greg Hogan <co...@greghogan.com> on 2016/02/08 19:34:18 UTC
Option to disable chaining?
Is it possible to force operator chaining to be disabled? Similar to how
object reuse can be enabled or disabled?
Greg
Re: Option to disable chaining?
Posted by Nick Dimiduk <nd...@gmail.com>.
Disabling operator chaining is also the only way to see metrics from each
step right now.
On Monday, February 8, 2016, Stephan Ewen <se...@apache.org> wrote:
> There is no fundamental reason to not implement this for batch as well.
>
> In Streaming, users seem to want more control about threads and resources
> (given that these are often continuous pipelines), while in batch that was
> not requested so far.
>
> But I see that a non-chained function is more safe with respect to
> accidental object reuse by a user in their functions...
>
> On Mon, Feb 8, 2016 at 8:19 PM, Greg Hogan <code@greghogan.com
> <javascript:;>> wrote:
>
> > Is there a reason to not also implement this for batch processing? This
> > would allow object reuse to be truly disabled.
> >
> > On Mon, Feb 8, 2016 at 2:02 PM, Stephan Ewen <sewen@apache.org
> <javascript:;>> wrote:
> >
> > > It may be useful for example when you have two MapFunctions and each
> does
> > > something CPU intensive, or communicates with an external service.
> > >
> > > Without chaining, you will have two threads and an elastic channel
> > between
> > > the functions to buffer some records, which may help in such a case.
> > >
> > >
> > > On Mon, Feb 8, 2016 at 7:57 PM, Greg Hogan <code@greghogan.com
> <javascript:;>> wrote:
> > >
> > > > When is this useful in streaming?
> > > >
> > > > On Mon, Feb 8, 2016 at 1:46 PM, Nick Dimiduk <ndimiduk@apache.org
> <javascript:;>>
> > > wrote:
> > > >
> > > > >
> > > > >
> > > >
> > >
> >
> https://ci.apache.org/projects/flink/flink-docs-release-0.10/api/java/org/apache/flink/streaming/api/environment/StreamExecutionEnvironment.html#disableOperatorChaining()
> > > > >
> > > > > On Mon, Feb 8, 2016 at 10:34 AM, Greg Hogan <code@greghogan.com
> <javascript:;>>
> > > wrote:
> > > > >
> > > > > > Is it possible to force operator chaining to be disabled? Similar
> > to
> > > > how
> > > > > > object reuse can be enabled or disabled?
> > > > > >
> > > > > > Greg
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Option to disable chaining?
Posted by Stephan Ewen <se...@apache.org>.
There is no fundamental reason to not implement this for batch as well.
In Streaming, users seem to want more control about threads and resources
(given that these are often continuous pipelines), while in batch that was
not requested so far.
But I see that a non-chained function is more safe with respect to
accidental object reuse by a user in their functions...
On Mon, Feb 8, 2016 at 8:19 PM, Greg Hogan <co...@greghogan.com> wrote:
> Is there a reason to not also implement this for batch processing? This
> would allow object reuse to be truly disabled.
>
> On Mon, Feb 8, 2016 at 2:02 PM, Stephan Ewen <se...@apache.org> wrote:
>
> > It may be useful for example when you have two MapFunctions and each does
> > something CPU intensive, or communicates with an external service.
> >
> > Without chaining, you will have two threads and an elastic channel
> between
> > the functions to buffer some records, which may help in such a case.
> >
> >
> > On Mon, Feb 8, 2016 at 7:57 PM, Greg Hogan <co...@greghogan.com> wrote:
> >
> > > When is this useful in streaming?
> > >
> > > On Mon, Feb 8, 2016 at 1:46 PM, Nick Dimiduk <nd...@apache.org>
> > wrote:
> > >
> > > >
> > > >
> > >
> >
> https://ci.apache.org/projects/flink/flink-docs-release-0.10/api/java/org/apache/flink/streaming/api/environment/StreamExecutionEnvironment.html#disableOperatorChaining()
> > > >
> > > > On Mon, Feb 8, 2016 at 10:34 AM, Greg Hogan <co...@greghogan.com>
> > wrote:
> > > >
> > > > > Is it possible to force operator chaining to be disabled? Similar
> to
> > > how
> > > > > object reuse can be enabled or disabled?
> > > > >
> > > > > Greg
> > > > >
> > > >
> > >
> >
>
Re: Option to disable chaining?
Posted by Greg Hogan <co...@greghogan.com>.
Is there a reason to not also implement this for batch processing? This
would allow object reuse to be truly disabled.
On Mon, Feb 8, 2016 at 2:02 PM, Stephan Ewen <se...@apache.org> wrote:
> It may be useful for example when you have two MapFunctions and each does
> something CPU intensive, or communicates with an external service.
>
> Without chaining, you will have two threads and an elastic channel between
> the functions to buffer some records, which may help in such a case.
>
>
> On Mon, Feb 8, 2016 at 7:57 PM, Greg Hogan <co...@greghogan.com> wrote:
>
> > When is this useful in streaming?
> >
> > On Mon, Feb 8, 2016 at 1:46 PM, Nick Dimiduk <nd...@apache.org>
> wrote:
> >
> > >
> > >
> >
> https://ci.apache.org/projects/flink/flink-docs-release-0.10/api/java/org/apache/flink/streaming/api/environment/StreamExecutionEnvironment.html#disableOperatorChaining()
> > >
> > > On Mon, Feb 8, 2016 at 10:34 AM, Greg Hogan <co...@greghogan.com>
> wrote:
> > >
> > > > Is it possible to force operator chaining to be disabled? Similar to
> > how
> > > > object reuse can be enabled or disabled?
> > > >
> > > > Greg
> > > >
> > >
> >
>
Re: Option to disable chaining?
Posted by Stephan Ewen <se...@apache.org>.
It may be useful for example when you have two MapFunctions and each does
something CPU intensive, or communicates with an external service.
Without chaining, you will have two threads and an elastic channel between
the functions to buffer some records, which may help in such a case.
On Mon, Feb 8, 2016 at 7:57 PM, Greg Hogan <co...@greghogan.com> wrote:
> When is this useful in streaming?
>
> On Mon, Feb 8, 2016 at 1:46 PM, Nick Dimiduk <nd...@apache.org> wrote:
>
> >
> >
> https://ci.apache.org/projects/flink/flink-docs-release-0.10/api/java/org/apache/flink/streaming/api/environment/StreamExecutionEnvironment.html#disableOperatorChaining()
> >
> > On Mon, Feb 8, 2016 at 10:34 AM, Greg Hogan <co...@greghogan.com> wrote:
> >
> > > Is it possible to force operator chaining to be disabled? Similar to
> how
> > > object reuse can be enabled or disabled?
> > >
> > > Greg
> > >
> >
>
Re: Option to disable chaining?
Posted by Greg Hogan <co...@greghogan.com>.
When is this useful in streaming?
On Mon, Feb 8, 2016 at 1:46 PM, Nick Dimiduk <nd...@apache.org> wrote:
>
> https://ci.apache.org/projects/flink/flink-docs-release-0.10/api/java/org/apache/flink/streaming/api/environment/StreamExecutionEnvironment.html#disableOperatorChaining()
>
> On Mon, Feb 8, 2016 at 10:34 AM, Greg Hogan <co...@greghogan.com> wrote:
>
> > Is it possible to force operator chaining to be disabled? Similar to how
> > object reuse can be enabled or disabled?
> >
> > Greg
> >
>
Re: Option to disable chaining?
Posted by Nick Dimiduk <nd...@apache.org>.
https://ci.apache.org/projects/flink/flink-docs-release-0.10/api/java/org/apache/flink/streaming/api/environment/StreamExecutionEnvironment.html#disableOperatorChaining()
On Mon, Feb 8, 2016 at 10:34 AM, Greg Hogan <co...@greghogan.com> wrote:
> Is it possible to force operator chaining to be disabled? Similar to how
> object reuse can be enabled or disabled?
>
> Greg
>