You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Jan Lukavský <je...@seznam.cz> on 2020/08/10 10:34:47 UTC
[DISCUSS] Better alignment of Apache Flink and Apache Beam releases
Hi,
I "sense a disturbance in the force" relating to the way we release Beam
with supported Flink versions. The current released version of Apache
Flink is 1.11.1, while we still support (at least up to Beam 2.24.0)
only version 1.10.1. There is tracking issue for 1.11. support [1], but
even if someone starts to work on this soon, it will probably not make
it to sooner release than 2.26.0, surely not before 2.25.0). I think
that the features included in newest Flink releases are pretty much
needed by our users, so I'd like to revive a somewhat left-over
discussion started in [2]. I think that we could be more aligned with
Flink's release is we created the following workflow:
- when a new Flink version is released, create a new branch for
flink-runner-<new_version>
- this new branch would depend on publihed SNAPSHOT version of the
not-yet-released version of Flink
- we would need a jenkins job that would periodically do builds
against new SNAPSHOTs and notify (some, volunteers welcome :))
committers about the status of the build
- this way, we might have people aware of incompatibilities, and
(pretty much) increase the chance, that the new runner branch would be
in shape to be able to switch from SNAPSHOT to release as soon as the
version of Beam gets released, merging the released version would mean
we create another branch for the new SNAPSHOT of Flink and repeat the
process
This workflow would rely on volunteer commiters (I'm one) that would be
willing to be notified about the failures and possibly fix them.
Looking forward for opinions, or alternative proposals to tackle this.
Jan
[1] https://issues.apache.org/jira/browse/BEAM-10612
[2]
https://lists.apache.org/thread.html/rfb5ac9d889d0e3f4400471de3c25000a15352bde879622c899d97581%40%3Cdev.beam.apache.org%3E
Re: [DISCUSS] Better alignment of Apache Flink and Apache Beam releases
Posted by Kenneth Knowles <ke...@apache.org>.
Agree with Luke's concerns about branches. We already have separate
subdirectories for Flink versions all on the master branch.
If we replace "branch" with "subdirectory of
https://github.com/apache/beam/tree/master/runners/flink" then we could
have a 1.12-SNAPSHOT variant of the runner which we do not release. This
would automatically catch new incompatibilities. Does Flink publish
snapshots regularly?
We can also create the 1.11 subdirectory at any time and only release it
when ready, no?
At some point our testing matrix will get infeasibly large, but we can
mitigate - for example by running more stable tests less frequently.
Kenn
On Mon, Aug 10, 2020 at 7:33 PM Luke Cwik <lc...@google.com> wrote:
> Is there a way we could use a fixed point in time Flink nightly that
> passes all the tests/validation and bump up the nightly version manually to
> get "closer" to the release candidate instead of doing another branch?
>
> This would mean that any changes that impact the Flink runner that are
> related to project clean-up or that are cross-cutting would be responsible
> to fix the nightly version as well. It might also lead to fewer
> integrations and merge conflicts when attempting to merge said branch back
> into master.
>
>
> On Mon, Aug 10, 2020 at 3:35 AM Jan Lukavský <je...@seznam.cz> wrote:
>
>> Hi,
>>
>> I "sense a disturbance in the force" relating to the way we release Beam
>> with supported Flink versions. The current released version of Apache
>> Flink is 1.11.1, while we still support (at least up to Beam 2.24.0)
>> only version 1.10.1. There is tracking issue for 1.11. support [1], but
>> even if someone starts to work on this soon, it will probably not make
>> it to sooner release than 2.26.0, surely not before 2.25.0). I think
>> that the features included in newest Flink releases are pretty much
>> needed by our users, so I'd like to revive a somewhat left-over
>> discussion started in [2]. I think that we could be more aligned with
>> Flink's release is we created the following workflow:
>>
>> - when a new Flink version is released, create a new branch for
>> flink-runner-<new_version>
>>
>> - this new branch would depend on publihed SNAPSHOT version of the
>> not-yet-released version of Flink
>>
>> - we would need a jenkins job that would periodically do builds
>> against new SNAPSHOTs and notify (some, volunteers welcome :))
>> committers about the status of the build
>>
>> - this way, we might have people aware of incompatibilities, and
>> (pretty much) increase the chance, that the new runner branch would be
>> in shape to be able to switch from SNAPSHOT to release as soon as the
>> version of Beam gets released, merging the released version would mean
>> we create another branch for the new SNAPSHOT of Flink and repeat the
>> process
>>
>> This workflow would rely on volunteer commiters (I'm one) that would be
>> willing to be notified about the failures and possibly fix them.
>>
>> Looking forward for opinions, or alternative proposals to tackle this.
>>
>> Jan
>>
>> [1] https://issues.apache.org/jira/browse/BEAM-10612
>>
>> [2]
>>
>> https://lists.apache.org/thread.html/rfb5ac9d889d0e3f4400471de3c25000a15352bde879622c899d97581%40%3Cdev.beam.apache.org%3E
>>
>>
Re: [DISCUSS] Better alignment of Apache Flink and Apache Beam releases
Posted by Luke Cwik <lc...@google.com>.
Is there a way we could use a fixed point in time Flink nightly that passes
all the tests/validation and bump up the nightly version manually to get
"closer" to the release candidate instead of doing another branch?
This would mean that any changes that impact the Flink runner that are
related to project clean-up or that are cross-cutting would be responsible
to fix the nightly version as well. It might also lead to fewer
integrations and merge conflicts when attempting to merge said branch back
into master.
On Mon, Aug 10, 2020 at 3:35 AM Jan Lukavský <je...@seznam.cz> wrote:
> Hi,
>
> I "sense a disturbance in the force" relating to the way we release Beam
> with supported Flink versions. The current released version of Apache
> Flink is 1.11.1, while we still support (at least up to Beam 2.24.0)
> only version 1.10.1. There is tracking issue for 1.11. support [1], but
> even if someone starts to work on this soon, it will probably not make
> it to sooner release than 2.26.0, surely not before 2.25.0). I think
> that the features included in newest Flink releases are pretty much
> needed by our users, so I'd like to revive a somewhat left-over
> discussion started in [2]. I think that we could be more aligned with
> Flink's release is we created the following workflow:
>
> - when a new Flink version is released, create a new branch for
> flink-runner-<new_version>
>
> - this new branch would depend on publihed SNAPSHOT version of the
> not-yet-released version of Flink
>
> - we would need a jenkins job that would periodically do builds
> against new SNAPSHOTs and notify (some, volunteers welcome :))
> committers about the status of the build
>
> - this way, we might have people aware of incompatibilities, and
> (pretty much) increase the chance, that the new runner branch would be
> in shape to be able to switch from SNAPSHOT to release as soon as the
> version of Beam gets released, merging the released version would mean
> we create another branch for the new SNAPSHOT of Flink and repeat the
> process
>
> This workflow would rely on volunteer commiters (I'm one) that would be
> willing to be notified about the failures and possibly fix them.
>
> Looking forward for opinions, or alternative proposals to tackle this.
>
> Jan
>
> [1] https://issues.apache.org/jira/browse/BEAM-10612
>
> [2]
>
> https://lists.apache.org/thread.html/rfb5ac9d889d0e3f4400471de3c25000a15352bde879622c899d97581%40%3Cdev.beam.apache.org%3E
>
>