You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Stephan Ewen <se...@apache.org> on 2018/05/11 18:41:07 UTC
[DISCUSS] Configuration for local recovery
Hi!
The configuration option (in flink-conf.yaml) for local recovery is
currently an enumeration with the values "DISABLED" and "ENABLE_FILE_BASED".
I would suggest to change that, for a few reasons:
- Having values like "ENABLE_FILE_BASED" breaks with the style of the
other config options. Having a homogeneous feel for the configuration of
the system is important for ease of use.
- Do we need to require users to understand what file-based local
recovery means? It might be easier for users to have an option to activate
deactivate the mode (on by default in the future) and if we need to have
different modes in the future, then we can have a "mode" option as an
"expert option". That way we expose the simple fact of whether to use local
recovery or not in a simple boolean, and hide the complex tuning part
(which hopefully few users ever need to touch) in a separate option.
- Are we sure already whether options beyond "on/off" are shared across
state backends? For example, memory snapshot based local recovery would be
specific to the Memoy/FsStateBackend. Persistent-volume based local
recovery may behave differently for RocksDB and FsStateBackend.
==> This config option looks like it sets things up in a tricky direction.
We can still change it, now that we have not yet released it.
Best,
Stephan
Re: [DISCUSS] Configuration for local recovery
Posted by Stefan Richter <s....@data-artisans.com>.
I think that is a good idea +1.
> Am 11.05.2018 um 20:41 schrieb Stephan Ewen <se...@apache.org>:
>
> Hi!
>
> The configuration option (in flink-conf.yaml) for local recovery is currently an enumeration with the values "DISABLED" and "ENABLE_FILE_BASED".
>
> I would suggest to change that, for a few reasons:
>
> - Having values like "ENABLE_FILE_BASED" breaks with the style of the other config options. Having a homogeneous feel for the configuration of the system is important for ease of use.
>
> - Do we need to require users to understand what file-based local recovery means? It might be easier for users to have an option to activate deactivate the mode (on by default in the future) and if we need to have different modes in the future, then we can have a "mode" option as an "expert option". That way we expose the simple fact of whether to use local recovery or not in a simple boolean, and hide the complex tuning part (which hopefully few users ever need to touch) in a separate option.
>
> - Are we sure already whether options beyond "on/off" are shared across state backends? For example, memory snapshot based local recovery would be specific to the Memoy/FsStateBackend. Persistent-volume based local recovery may behave differently for RocksDB and FsStateBackend.
>
>
> ==> This config option looks like it sets things up in a tricky direction. We can still change it, now that we have not yet released it.
>
> Best,
> Stephan
>
Re: [DISCUSS] Configuration for local recovery
Posted by Fabian Hueske <fh...@gmail.com>.
+1 for Stephan's proposal.
2018-05-14 8:22 GMT+02:00 Shuyi Chen <su...@gmail.com>:
> +1 to the proposal. IMO, the current option "ENABLE_FILE_BASED" contains
> too much implementation details and might confuse the simple users. Having
> a simple on/off toggle for majority of the users and an advanced option for
> the experts to do the tuning will definitely make better user experience.
>
> On Sun, May 13, 2018 at 12:33 PM, Till Rohrmann <tr...@apache.org>
> wrote:
>
> > I agree with Stephan that a simple on/off configuration option for local
> > recovery would be easier to understand and gives more flexibility wrt
> > future changes.
> >
> > Cheers,
> > Till
> >
> > On Sun, May 13, 2018 at 4:00 PM, sihua zhou <su...@163.com> wrote:
> >
> > > +1 for @Stephan's proposal, it makes the out of the box experience
> better
> > > and also leaves some space for the expert.
> > >
> > > Best,
> > > Sihua
> > >
> > >
> > >
> > > On 05/12/2018 02:41，Stephan Ewen<se...@apache.org> <se...@apache.org>
> > > wrote：
> > >
> > > Hi!
> > >
> > > The configuration option (in flink-conf.yaml) for local recovery is
> > > currently an enumeration with the values "DISABLED" and
> > > "ENABLE_FILE_BASED".
> > >
> > > I would suggest to change that, for a few reasons:
> > >
> > > - Having values like "ENABLE_FILE_BASED" breaks with the style of the
> > > other config options. Having a homogeneous feel for the configuration
> of
> > > the system is important for ease of use.
> > >
> > > - Do we need to require users to understand what file-based local
> > > recovery means? It might be easier for users to have an option to
> > activate
> > > deactivate the mode (on by default in the future) and if we need to
> have
> > > different modes in the future, then we can have a "mode" option as an
> > > "expert option". That way we expose the simple fact of whether to use
> > local
> > > recovery or not in a simple boolean, and hide the complex tuning part
> > > (which hopefully few users ever need to touch) in a separate option.
> > >
> > > - Are we sure already whether options beyond "on/off" are shared across
> > > state backends? For example, memory snapshot based local recovery would
> > be
> > > specific to the Memoy/FsStateBackend. Persistent-volume based local
> > > recovery may behave differently for RocksDB and FsStateBackend.
> > >
> > >
> > > ==> This config option looks like it sets things up in a tricky
> > direction.
> > > We can still change it, now that we have not yet released it.
> > >
> > > Best,
> > > Stephan
> > >
> > >
> >
>
>
>
> --
> "So you have to trust that the dots will somehow connect in your future."
>
Re: [DISCUSS] Configuration for local recovery
Posted by Shuyi Chen <su...@gmail.com>.
+1 to the proposal. IMO, the current option "ENABLE_FILE_BASED" contains
too much implementation details and might confuse the simple users. Having
a simple on/off toggle for majority of the users and an advanced option for
the experts to do the tuning will definitely make better user experience.
On Sun, May 13, 2018 at 12:33 PM, Till Rohrmann <tr...@apache.org>
wrote:
> I agree with Stephan that a simple on/off configuration option for local
> recovery would be easier to understand and gives more flexibility wrt
> future changes.
>
> Cheers,
> Till
>
> On Sun, May 13, 2018 at 4:00 PM, sihua zhou <su...@163.com> wrote:
>
> > +1 for @Stephan's proposal, it makes the out of the box experience better
> > and also leaves some space for the expert.
> >
> > Best,
> > Sihua
> >
> >
> >
> > On 05/12/2018 02:41，Stephan Ewen<se...@apache.org> <se...@apache.org>
> > wrote：
> >
> > Hi!
> >
> > The configuration option (in flink-conf.yaml) for local recovery is
> > currently an enumeration with the values "DISABLED" and
> > "ENABLE_FILE_BASED".
> >
> > I would suggest to change that, for a few reasons:
> >
> > - Having values like "ENABLE_FILE_BASED" breaks with the style of the
> > other config options. Having a homogeneous feel for the configuration of
> > the system is important for ease of use.
> >
> > - Do we need to require users to understand what file-based local
> > recovery means? It might be easier for users to have an option to
> activate
> > deactivate the mode (on by default in the future) and if we need to have
> > different modes in the future, then we can have a "mode" option as an
> > "expert option". That way we expose the simple fact of whether to use
> local
> > recovery or not in a simple boolean, and hide the complex tuning part
> > (which hopefully few users ever need to touch) in a separate option.
> >
> > - Are we sure already whether options beyond "on/off" are shared across
> > state backends? For example, memory snapshot based local recovery would
> be
> > specific to the Memoy/FsStateBackend. Persistent-volume based local
> > recovery may behave differently for RocksDB and FsStateBackend.
> >
> >
> > ==> This config option looks like it sets things up in a tricky
> direction.
> > We can still change it, now that we have not yet released it.
> >
> > Best,
> > Stephan
> >
> >
>
--
"So you have to trust that the dots will somehow connect in your future."
Re: [DISCUSS] Configuration for local recovery
Posted by Till Rohrmann <tr...@apache.org>.
I agree with Stephan that a simple on/off configuration option for local
recovery would be easier to understand and gives more flexibility wrt
future changes.
Cheers,
Till
On Sun, May 13, 2018 at 4:00 PM, sihua zhou <su...@163.com> wrote:
> +1 for @Stephan's proposal, it makes the out of the box experience better
> and also leaves some space for the expert.
>
> Best,
> Sihua
>
>
>
> On 05/12/2018 02:41，Stephan Ewen<se...@apache.org> <se...@apache.org>
> wrote：
>
> Hi!
>
> The configuration option (in flink-conf.yaml) for local recovery is
> currently an enumeration with the values "DISABLED" and
> "ENABLE_FILE_BASED".
>
> I would suggest to change that, for a few reasons:
>
> - Having values like "ENABLE_FILE_BASED" breaks with the style of the
> other config options. Having a homogeneous feel for the configuration of
> the system is important for ease of use.
>
> - Do we need to require users to understand what file-based local
> recovery means? It might be easier for users to have an option to activate
> deactivate the mode (on by default in the future) and if we need to have
> different modes in the future, then we can have a "mode" option as an
> "expert option". That way we expose the simple fact of whether to use local
> recovery or not in a simple boolean, and hide the complex tuning part
> (which hopefully few users ever need to touch) in a separate option.
>
> - Are we sure already whether options beyond "on/off" are shared across
> state backends? For example, memory snapshot based local recovery would be
> specific to the Memoy/FsStateBackend. Persistent-volume based local
> recovery may behave differently for RocksDB and FsStateBackend.
>
>
> ==> This config option looks like it sets things up in a tricky direction.
> We can still change it, now that we have not yet released it.
>
> Best,
> Stephan
>
>