You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Greg Hogan <co...@greghogan.com> on 2017/02/06 18:21:21 UTC
[Discuss] Organizing Documentation for Configuration Options
Hi devs,
Flink's Configuration page [1] has grown intimidatingly long and complex.
Options are described across three main sections: common options (single
section), advanced options (multiple sections), and full reference. The
trailing "background" section further describes the most impactful options
in much greater detail.
Several recent tickets, and a few outstanding, have added missing options
to the configuration documentation. I'd like to propose a goal of
organizing all options in the full reference into alphabetized, tabular
form (one table per section), much like the system metrics [2]. Columns
would be option name, description, and default value.
The common and advanced sections could also be converted to tabular form
with the exception of Kerberos-based Security. Missing options would be
added to the full reference.
Lastly, the simple heuristic for configuring network buffers has prompted
many questions on the mailing list. With the 1.3 release the total and
number of available buffers is reported through metrics and in the web
dashboard. My experience has been that the number of required buffers is
highly dependent on job topology and cluster performance. I propose keeping
the simple heuristic and description while directing users to monitor the
balance of available buffers.
Greg
[1] https://ci.apache.org/projects/flink/flink-docs-master/setup/config.html
[2]
https://ci.apache.org/projects/flink/flink-docs-master/monitoring/metrics.html#system-metrics
[3]
https://ci.apache.org/projects/flink/flink-docs-master/setup/config.html#configuring-the-network-buffers
Re: [Discuss] Organizing Documentation for Configuration Options
Posted by Ufuk Celebi <uc...@apache.org>.
Unfortunately, I didn't have time to work on this yet and it looks
like I won't have time this week.
I've created the following issue to track the implementation with two sub tasks:
https://issues.apache.org/jira/browse/FLINK-5779
If someone else wants to take this up in the meantime, feel free to do
so. Otherwise, I will try to have a look next week. I can certainly
help with the initial setup if there are questions.
On Tue, Feb 7, 2017 at 3:14 PM, Greg Hogan <co...@greghogan.com> wrote:
> +1 and thanks for volunteering for an initial implementation.
>
> Looking forward to auto-scaling of network buffers.
>
> On Tue, Feb 7, 2017 at 3:04 AM, Ufuk Celebi <uc...@apache.org> wrote:
>
>> I fully agree with you Greg.
>>
>> Since this is doomed to get out of sync again very shortly after clean up,
>> I vote to automate this. Stephan introduced the ConfigOption type, which
>> makes it easy to define the options. It's already planned to migrate all
>> configuration options from ConfigConstants to this approach.
>>
>> For an example see here: https://github.com/apache/flink/blob/master/
>> flink-core/src/main/java/org/apache/flink/configuration/
>> HighAvailabilityOptions.java
>>
>> I think that it is possible to build the configuration docs page from this
>> with reasonable effort.
>>
>> This would translate the task to:
>> 1) Automate ConfigOption to HTML/Markdown generation
>> 2) Extend ConfigOption with description fields
>> 3) Migrate ConfigConstants to ConfigOptions
>>
>> I would also volunteer to take a first stab at this.
>>
>> Regarding the network buffers: +1 to your suggestion. Nico (cc'd) is
>> starting to work on automating the network buffer configuration in order to
>> get rid of any manual tuning for most users (because of the issues you
>> described + streaming and batch jobs require different tuning, which
>> complicates things even more).
>>
>> – Ufuk
>>
>> On 6 February 2017 at 19:21:28, Greg Hogan (code@greghogan.com) wrote:
>> > > Hi devs,
>> >
>> > Flink's Configuration page [1] has grown intimidatingly long
>> > and complex.
>> > Options are described across three main sections: common options
>> > (single
>> > section), advanced options (multiple sections), and full reference.
>> > The
>> > trailing "background" section further describes the most impactful
>> > options
>> > in much greater detail.
>> >
>> > Several recent tickets, and a few outstanding, have added missing
>> > options
>> > to the configuration documentation. I'd like to propose a goal
>> > of
>> > organizing all options in the full reference into alphabetized,
>> > tabular
>> > form (one table per section), much like the system metrics [2].
>> > Columns
>> > would be option name, description, and default value.
>> >
>> > The common and advanced sections could also be converted to tabular
>> > form
>> > with the exception of Kerberos-based Security. Missing options
>> > would be
>> > added to the full reference.
>> >
>> > Lastly, the simple heuristic for configuring network buffers
>> > has prompted
>> > many questions on the mailing list. With the 1.3 release the total
>> > and
>> > number of available buffers is reported through metrics and
>> > in the web
>> > dashboard. My experience has been that the number of required
>> > buffers is
>> > highly dependent on job topology and cluster performance. I
>> > propose keeping
>> > the simple heuristic and description while directing users
>> > to monitor the
>> > balance of available buffers.
>> >
>> > Greg
>> >
>> > [1] https://ci.apache.org/projects/flink/flink-docs-
>> master/setup/config.html
>> > [2]
>> > https://ci.apache.org/projects/flink/flink-docs-
>> master/monitoring/metrics.html#system-metrics
>> > [3]
>> > https://ci.apache.org/projects/flink/flink-docs-
>> master/setup/config.html#configuring-the-network-buffers
>>
>>
Re: [Discuss] Organizing Documentation for Configuration Options
Posted by Greg Hogan <co...@greghogan.com>.
+1 and thanks for volunteering for an initial implementation.
Looking forward to auto-scaling of network buffers.
On Tue, Feb 7, 2017 at 3:04 AM, Ufuk Celebi <uc...@apache.org> wrote:
> I fully agree with you Greg.
>
> Since this is doomed to get out of sync again very shortly after clean up,
> I vote to automate this. Stephan introduced the ConfigOption type, which
> makes it easy to define the options. It's already planned to migrate all
> configuration options from ConfigConstants to this approach.
>
> For an example see here: https://github.com/apache/flink/blob/master/
> flink-core/src/main/java/org/apache/flink/configuration/
> HighAvailabilityOptions.java
>
> I think that it is possible to build the configuration docs page from this
> with reasonable effort.
>
> This would translate the task to:
> 1) Automate ConfigOption to HTML/Markdown generation
> 2) Extend ConfigOption with description fields
> 3) Migrate ConfigConstants to ConfigOptions
>
> I would also volunteer to take a first stab at this.
>
> Regarding the network buffers: +1 to your suggestion. Nico (cc'd) is
> starting to work on automating the network buffer configuration in order to
> get rid of any manual tuning for most users (because of the issues you
> described + streaming and batch jobs require different tuning, which
> complicates things even more).
>
> – Ufuk
>
> On 6 February 2017 at 19:21:28, Greg Hogan (code@greghogan.com) wrote:
> > > Hi devs,
> >
> > Flink's Configuration page [1] has grown intimidatingly long
> > and complex.
> > Options are described across three main sections: common options
> > (single
> > section), advanced options (multiple sections), and full reference.
> > The
> > trailing "background" section further describes the most impactful
> > options
> > in much greater detail.
> >
> > Several recent tickets, and a few outstanding, have added missing
> > options
> > to the configuration documentation. I'd like to propose a goal
> > of
> > organizing all options in the full reference into alphabetized,
> > tabular
> > form (one table per section), much like the system metrics [2].
> > Columns
> > would be option name, description, and default value.
> >
> > The common and advanced sections could also be converted to tabular
> > form
> > with the exception of Kerberos-based Security. Missing options
> > would be
> > added to the full reference.
> >
> > Lastly, the simple heuristic for configuring network buffers
> > has prompted
> > many questions on the mailing list. With the 1.3 release the total
> > and
> > number of available buffers is reported through metrics and
> > in the web
> > dashboard. My experience has been that the number of required
> > buffers is
> > highly dependent on job topology and cluster performance. I
> > propose keeping
> > the simple heuristic and description while directing users
> > to monitor the
> > balance of available buffers.
> >
> > Greg
> >
> > [1] https://ci.apache.org/projects/flink/flink-docs-
> master/setup/config.html
> > [2]
> > https://ci.apache.org/projects/flink/flink-docs-
> master/monitoring/metrics.html#system-metrics
> > [3]
> > https://ci.apache.org/projects/flink/flink-docs-
> master/setup/config.html#configuring-the-network-buffers
>
>
Re: [Discuss] Organizing Documentation for Configuration Options
Posted by Till Rohrmann <tr...@apache.org>.
+1 for automating. The more automation the fewer mistakes we can make :-)
On Tue, Feb 7, 2017 at 11:13 AM, Robert Metzger <rm...@apache.org> wrote:
> +1 to automate this and describe the config parameters in the code.
>
> That's exactly the approach Apache Kafka is taking as well for their
> config.
>
> On Tue, Feb 7, 2017 at 9:04 AM, Ufuk Celebi <uc...@apache.org> wrote:
>
> > I fully agree with you Greg.
> >
> > Since this is doomed to get out of sync again very shortly after clean
> up,
> > I vote to automate this. Stephan introduced the ConfigOption type, which
> > makes it easy to define the options. It's already planned to migrate all
> > configuration options from ConfigConstants to this approach.
> >
> > For an example see here: https://github.com/apache/flink/blob/master/
> > flink-core/src/main/java/org/apache/flink/configuration/
> > HighAvailabilityOptions.java
> >
> > I think that it is possible to build the configuration docs page from
> this
> > with reasonable effort.
> >
> > This would translate the task to:
> > 1) Automate ConfigOption to HTML/Markdown generation
> > 2) Extend ConfigOption with description fields
> > 3) Migrate ConfigConstants to ConfigOptions
> >
> > I would also volunteer to take a first stab at this.
> >
> > Regarding the network buffers: +1 to your suggestion. Nico (cc'd) is
> > starting to work on automating the network buffer configuration in order
> to
> > get rid of any manual tuning for most users (because of the issues you
> > described + streaming and batch jobs require different tuning, which
> > complicates things even more).
> >
> > – Ufuk
> >
> > On 6 February 2017 at 19:21:28, Greg Hogan (code@greghogan.com) wrote:
> > > > Hi devs,
> > >
> > > Flink's Configuration page [1] has grown intimidatingly long
> > > and complex.
> > > Options are described across three main sections: common options
> > > (single
> > > section), advanced options (multiple sections), and full reference.
> > > The
> > > trailing "background" section further describes the most impactful
> > > options
> > > in much greater detail.
> > >
> > > Several recent tickets, and a few outstanding, have added missing
> > > options
> > > to the configuration documentation. I'd like to propose a goal
> > > of
> > > organizing all options in the full reference into alphabetized,
> > > tabular
> > > form (one table per section), much like the system metrics [2].
> > > Columns
> > > would be option name, description, and default value.
> > >
> > > The common and advanced sections could also be converted to tabular
> > > form
> > > with the exception of Kerberos-based Security. Missing options
> > > would be
> > > added to the full reference.
> > >
> > > Lastly, the simple heuristic for configuring network buffers
> > > has prompted
> > > many questions on the mailing list. With the 1.3 release the total
> > > and
> > > number of available buffers is reported through metrics and
> > > in the web
> > > dashboard. My experience has been that the number of required
> > > buffers is
> > > highly dependent on job topology and cluster performance. I
> > > propose keeping
> > > the simple heuristic and description while directing users
> > > to monitor the
> > > balance of available buffers.
> > >
> > > Greg
> > >
> > > [1] https://ci.apache.org/projects/flink/flink-docs-
> > master/setup/config.html
> > > [2]
> > > https://ci.apache.org/projects/flink/flink-docs-
> > master/monitoring/metrics.html#system-metrics
> > > [3]
> > > https://ci.apache.org/projects/flink/flink-docs-
> > master/setup/config.html#configuring-the-network-buffers
> >
> >
>
Re: [Discuss] Organizing Documentation for Configuration Options
Posted by Robert Metzger <rm...@apache.org>.
+1 to automate this and describe the config parameters in the code.
That's exactly the approach Apache Kafka is taking as well for their config.
On Tue, Feb 7, 2017 at 9:04 AM, Ufuk Celebi <uc...@apache.org> wrote:
> I fully agree with you Greg.
>
> Since this is doomed to get out of sync again very shortly after clean up,
> I vote to automate this. Stephan introduced the ConfigOption type, which
> makes it easy to define the options. It's already planned to migrate all
> configuration options from ConfigConstants to this approach.
>
> For an example see here: https://github.com/apache/flink/blob/master/
> flink-core/src/main/java/org/apache/flink/configuration/
> HighAvailabilityOptions.java
>
> I think that it is possible to build the configuration docs page from this
> with reasonable effort.
>
> This would translate the task to:
> 1) Automate ConfigOption to HTML/Markdown generation
> 2) Extend ConfigOption with description fields
> 3) Migrate ConfigConstants to ConfigOptions
>
> I would also volunteer to take a first stab at this.
>
> Regarding the network buffers: +1 to your suggestion. Nico (cc'd) is
> starting to work on automating the network buffer configuration in order to
> get rid of any manual tuning for most users (because of the issues you
> described + streaming and batch jobs require different tuning, which
> complicates things even more).
>
> – Ufuk
>
> On 6 February 2017 at 19:21:28, Greg Hogan (code@greghogan.com) wrote:
> > > Hi devs,
> >
> > Flink's Configuration page [1] has grown intimidatingly long
> > and complex.
> > Options are described across three main sections: common options
> > (single
> > section), advanced options (multiple sections), and full reference.
> > The
> > trailing "background" section further describes the most impactful
> > options
> > in much greater detail.
> >
> > Several recent tickets, and a few outstanding, have added missing
> > options
> > to the configuration documentation. I'd like to propose a goal
> > of
> > organizing all options in the full reference into alphabetized,
> > tabular
> > form (one table per section), much like the system metrics [2].
> > Columns
> > would be option name, description, and default value.
> >
> > The common and advanced sections could also be converted to tabular
> > form
> > with the exception of Kerberos-based Security. Missing options
> > would be
> > added to the full reference.
> >
> > Lastly, the simple heuristic for configuring network buffers
> > has prompted
> > many questions on the mailing list. With the 1.3 release the total
> > and
> > number of available buffers is reported through metrics and
> > in the web
> > dashboard. My experience has been that the number of required
> > buffers is
> > highly dependent on job topology and cluster performance. I
> > propose keeping
> > the simple heuristic and description while directing users
> > to monitor the
> > balance of available buffers.
> >
> > Greg
> >
> > [1] https://ci.apache.org/projects/flink/flink-docs-
> master/setup/config.html
> > [2]
> > https://ci.apache.org/projects/flink/flink-docs-
> master/monitoring/metrics.html#system-metrics
> > [3]
> > https://ci.apache.org/projects/flink/flink-docs-
> master/setup/config.html#configuring-the-network-buffers
>
>
Re: [Discuss] Organizing Documentation for Configuration
Options
Posted by Ufuk Celebi <uc...@apache.org>.
I fully agree with you Greg.
Since this is doomed to get out of sync again very shortly after clean up, I vote to automate this. Stephan introduced the ConfigOption type, which makes it easy to define the options. It's already planned to migrate all configuration options from ConfigConstants to this approach.
For an example see here: https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/configuration/HighAvailabilityOptions.java
I think that it is possible to build the configuration docs page from this with reasonable effort.
This would translate the task to:
1) Automate ConfigOption to HTML/Markdown generation
2) Extend ConfigOption with description fields
3) Migrate ConfigConstants to ConfigOptions
I would also volunteer to take a first stab at this.
Regarding the network buffers: +1 to your suggestion. Nico (cc'd) is starting to work on automating the network buffer configuration in order to get rid of any manual tuning for most users (because of the issues you described + streaming and batch jobs require different tuning, which complicates things even more).
– Ufuk
On 6 February 2017 at 19:21:28, Greg Hogan (code@greghogan.com) wrote:
> > Hi devs,
>
> Flink's Configuration page [1] has grown intimidatingly long
> and complex.
> Options are described across three main sections: common options
> (single
> section), advanced options (multiple sections), and full reference.
> The
> trailing "background" section further describes the most impactful
> options
> in much greater detail.
>
> Several recent tickets, and a few outstanding, have added missing
> options
> to the configuration documentation. I'd like to propose a goal
> of
> organizing all options in the full reference into alphabetized,
> tabular
> form (one table per section), much like the system metrics [2].
> Columns
> would be option name, description, and default value.
>
> The common and advanced sections could also be converted to tabular
> form
> with the exception of Kerberos-based Security. Missing options
> would be
> added to the full reference.
>
> Lastly, the simple heuristic for configuring network buffers
> has prompted
> many questions on the mailing list. With the 1.3 release the total
> and
> number of available buffers is reported through metrics and
> in the web
> dashboard. My experience has been that the number of required
> buffers is
> highly dependent on job topology and cluster performance. I
> propose keeping
> the simple heuristic and description while directing users
> to monitor the
> balance of available buffers.
>
> Greg
>
> [1] https://ci.apache.org/projects/flink/flink-docs-master/setup/config.html
> [2]
> https://ci.apache.org/projects/flink/flink-docs-master/monitoring/metrics.html#system-metrics
> [3]
> https://ci.apache.org/projects/flink/flink-docs-master/setup/config.html#configuring-the-network-buffers