[DISCUSS] Move docker development into versioned branches

[Chesnay Schepler <ch...@apache.org>]

Hello everyone,

Currently, all development in docker-flink occurs on the master branch, 
for all releases at once. The master branch also serves as a publishing 
area for our docker files.

This means that all versions share the same generators, templates (i.e., 
production code that we plugin in some variables like the Flink version) 
and test setup.

So far this worked fine because all versions worked the same, but we are 
now coming to a point where we are adding version-specific features, 
like support for Java 11 (FLINK-16260) and per-job-clusters (FLINK-17164).

By virtue of all files being shared this means that we would have to 
introduce version checks in both the generators, templates and test 
code, so things continue to work for all versions. This is rather 
painful to do, and presents maintenance problems as these conditions 
must of course be updated at some point, and diverging behaviors require 
all versions to be tested on each PR (in contrast to the current state 
where we only test 1 version).

It is also simply different to the process we're using for 
flink/flink-shaded, creating an additional entry barrier.

I propose moving the development of the docker files into dedicated 
branches (dev-master, dev-1.10, dev-1.9). PullRequests will usually be 
opened against dev-master, possibly ported to other versions, and when 
it is time for a release we will update the master branch with the 
latest dockerfiles. Quite similar to how things work in the Flink repo.

The master branch would continue to be our release publishing area to 
stick to docker conventions.

Regards,

Chesnay

Re: [DISCUSS] Move docker development into versioned branches

[Patrick Lucas <pa...@ververica.com>]

This also sounds good to me. The master branch will continue to be the
publishing area for the release Dockerfiles and we can support changes
between Flink versions without additional complexity.

My only comment would be that I hope we can still keep eventual convergence
in mind—it should be possible, with some changes in Flink, to make the
differences in the Docker packaging between versions negligible, to the
point that this split might stop being necessary.

On Mon, Apr 27, 2020 at 3:07 PM Till Rohrmann <tr...@apache.org> wrote:

> Thanks for starting this discussion Chesnay. Your proposal sounds good to
> me. I can see how the current setup makes the development of version
> specific features impractical. Hence, +1 for the proposed changes.
>
> Cheers,
> Till
>
> On Mon, Apr 27, 2020 at 12:19 PM David Anderson <da...@alpinegizmo.com>
> wrote:
>
> > Makes sense to me. I think this would align well enough with user
> > expectations, and be more straightforward.
> >
> > David
> >
> > On Thu, Apr 23, 2020 at 1:23 PM Chesnay Schepler <ch...@apache.org>
> > wrote:
> >
> > > Hello everyone,
> > >
> > > Currently, all development in docker-flink occurs on the master branch,
> > > for all releases at once. The master branch also serves as a publishing
> > > area for our docker files.
> > >
> > > This means that all versions share the same generators, templates
> (i.e.,
> > > production code that we plugin in some variables like the Flink
> version)
> > > and test setup.
> > >
> > > So far this worked fine because all versions worked the same, but we
> are
> > > now coming to a point where we are adding version-specific features,
> > > like support for Java 11 (FLINK-16260) and per-job-clusters
> > (FLINK-17164).
> > >
> > > By virtue of all files being shared this means that we would have to
> > > introduce version checks in both the generators, templates and test
> > > code, so things continue to work for all versions. This is rather
> > > painful to do, and presents maintenance problems as these conditions
> > > must of course be updated at some point, and diverging behaviors
> require
> > > all versions to be tested on each PR (in contrast to the current state
> > > where we only test 1 version).
> > >
> > > It is also simply different to the process we're using for
> > > flink/flink-shaded, creating an additional entry barrier.
> > >
> > > I propose moving the development of the docker files into dedicated
> > > branches (dev-master, dev-1.10, dev-1.9). PullRequests will usually be
> > > opened against dev-master, possibly ported to other versions, and when
> > > it is time for a release we will update the master branch with the
> > > latest dockerfiles. Quite similar to how things work in the Flink repo.
> > >
> > > The master branch would continue to be our release publishing area to
> > > stick to docker conventions.
> > >
> > > Regards,
> > >
> > > Chesnay
> > >
> > >
> >
>

Re: [DISCUSS] Move docker development into versioned branches

[Till Rohrmann <tr...@apache.org>]

Thanks for starting this discussion Chesnay. Your proposal sounds good to
me. I can see how the current setup makes the development of version
specific features impractical. Hence, +1 for the proposed changes.

Cheers,
Till

On Mon, Apr 27, 2020 at 12:19 PM David Anderson <da...@alpinegizmo.com>
wrote:

> Makes sense to me. I think this would align well enough with user
> expectations, and be more straightforward.
>
> David
>
> On Thu, Apr 23, 2020 at 1:23 PM Chesnay Schepler <ch...@apache.org>
> wrote:
>
> > Hello everyone,
> >
> > Currently, all development in docker-flink occurs on the master branch,
> > for all releases at once. The master branch also serves as a publishing
> > area for our docker files.
> >
> > This means that all versions share the same generators, templates (i.e.,
> > production code that we plugin in some variables like the Flink version)
> > and test setup.
> >
> > So far this worked fine because all versions worked the same, but we are
> > now coming to a point where we are adding version-specific features,
> > like support for Java 11 (FLINK-16260) and per-job-clusters
> (FLINK-17164).
> >
> > By virtue of all files being shared this means that we would have to
> > introduce version checks in both the generators, templates and test
> > code, so things continue to work for all versions. This is rather
> > painful to do, and presents maintenance problems as these conditions
> > must of course be updated at some point, and diverging behaviors require
> > all versions to be tested on each PR (in contrast to the current state
> > where we only test 1 version).
> >
> > It is also simply different to the process we're using for
> > flink/flink-shaded, creating an additional entry barrier.
> >
> > I propose moving the development of the docker files into dedicated
> > branches (dev-master, dev-1.10, dev-1.9). PullRequests will usually be
> > opened against dev-master, possibly ported to other versions, and when
> > it is time for a release we will update the master branch with the
> > latest dockerfiles. Quite similar to how things work in the Flink repo.
> >
> > The master branch would continue to be our release publishing area to
> > stick to docker conventions.
> >
> > Regards,
> >
> > Chesnay
> >
> >
>

Re: [DISCUSS] Move docker development into versioned branches

[David Anderson <da...@alpinegizmo.com>]

Makes sense to me. I think this would align well enough with user
expectations, and be more straightforward.

David

On Thu, Apr 23, 2020 at 1:23 PM Chesnay Schepler <ch...@apache.org> wrote:

> Hello everyone,
>
> Currently, all development in docker-flink occurs on the master branch,
> for all releases at once. The master branch also serves as a publishing
> area for our docker files.
>
> This means that all versions share the same generators, templates (i.e.,
> production code that we plugin in some variables like the Flink version)
> and test setup.
>
> So far this worked fine because all versions worked the same, but we are
> now coming to a point where we are adding version-specific features,
> like support for Java 11 (FLINK-16260) and per-job-clusters (FLINK-17164).
>
> By virtue of all files being shared this means that we would have to
> introduce version checks in both the generators, templates and test
> code, so things continue to work for all versions. This is rather
> painful to do, and presents maintenance problems as these conditions
> must of course be updated at some point, and diverging behaviors require
> all versions to be tested on each PR (in contrast to the current state
> where we only test 1 version).
>
> It is also simply different to the process we're using for
> flink/flink-shaded, creating an additional entry barrier.
>
> I propose moving the development of the docker files into dedicated
> branches (dev-master, dev-1.10, dev-1.9). PullRequests will usually be
> opened against dev-master, possibly ported to other versions, and when
> it is time for a release we will update the master branch with the
> latest dockerfiles. Quite similar to how things work in the Flink repo.
>
> The master branch would continue to be our release publishing area to
> stick to docker conventions.
>
> Regards,
>
> Chesnay
>
>