You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Chesnay Schepler <ch...@apache.org> on 2021/04/12 11:17:16 UTC
[DISCUSS] Flink should rely on flink-shaded SNAPSHOT artifacts
Hello all,
I would like to propose a change in how the Flink master interacts with
Flink-shaded.
TL;DR: Release snapshot artifacts for flink-shaded, and have the Flink
master rely on specific snapshot versions for earlier dependency bumps.
Aa a project we have come to the general conclusion that dependencies
should be bumped as early in the release cycle as possible. This both
prevents cases where some undefined amount of work is still waiting for
as when we want to release the next version (working against the goal of
always being in a releasable state), and it gives us more time to
evaluate the stability and performance of system. Finally it gives us
ample time to look for alternatives if an issue is found.
Currently, this conclusion is at odds with how we handle flink-shaded.
Flink has always relied on flink-shaded artifacts that went through a
proper release cycle. However, since we want to create as few releases
as possible due to the overhead/noise/etc., flink-shaded releases are
typically relegated to the end of the release cycle.
This is particularly troublesome since flink-shaded dependencies are
used in the core of Flink, and hence usage of them cannot be avoided.
As a compromise between these 2 goals I propose the following:
- we deploy SNAPSHOT artifacts for flink-shaded for every change made
- every deployed artifact has a unique version, that is automatically
set via maven (=> no overhead on our side)
- once such an artifact is released we update the Flink dependency to
point to this _specific_ flink-shaded snapshot artifact
- to be clear, this is a manual step, which implies that things
cannot break all of a sudden because something was pushed to flink-shaded
- once the Flink release cycle ends, we publish a proper flink-shaded
release, and change the Flink dependency in the release branch accordingly
This should give us the best of both worlds: We have as few releases as
necessary (at most 1 per Flink release cycle), but can update the
dependencies in Flink as soon as possible.
Furthermore, this can also be considered a test run for how multiple
repos with the same release cycle could be developed in sync with each
other.
Let me know what you think.
Regards,
Chesnay
Re: [DISCUSS] Flink should rely on flink-shaded SNAPSHOT artifacts
Posted by Chesnay Schepler <ch...@apache.org>.
@Robert We can workaround the snapshot limit issue fairly easily; this
limit is imposed per version, so if we modify the version to include the
commit hash this limit does not apply. This should also make it easier
to work with from the Flink side because a commit hash is easier to
copy&paste than figuring out which timestamp you need.
@Till Yes, the release process involves too many manual steps, and I
don't want to spend time on it unless I'm sure we don't need another
one. Many of these cannot be automated AFAIK, like the book-keeping at
reporter.apache.org, preparing and handling the voting threads,
preparing the flink-web PR, managing the snapshot repository, and so on.
As for removing the SNAPSHOT version, we can enable an enforcer rule to
forbid snapshot dependencies. It's a good safeguard in any case. We can
furthermore automate the removal, and do it as part of the release
branch creation.
On 4/13/2021 10:47 AM, Till Rohrmann wrote:
> Thanks for creating this proposal Chesnay. I do understand the problem
> you want to fix.
>
> What I am wondering is why we don't release flink-shaded more often.
> Does the release process cause too much overhead? If this is the case,
> then we could look into what is causing the overhead and whether we
> can improve the situation. Concerning the noise, I personally don't
> see it as a problem.
>
> My main concern is that it can easily slip our minds to change the
> flink-shaded SNAPSHOT version to a non SNAPSHOT version and that it
> introduces another manual step. If we forget to change the version,
> then the Flink release does not build against a stable set of
> dependencies. Moreover, I also second Robert's concern that a single
> commit to flink-shaded can then break downstream projects (Flink in
> this case) if we rely on the SNAPSHOT builds. Having to scan poms for
> some references sounds like an indicator that this might not be the
> most straight forward approach.
>
> Cheers,
> Till
>
> On Tue, Apr 13, 2021 at 9:26 AM Robert Metzger <rmetzger@apache.org
> <ma...@apache.org>> wrote:
>
> Thanks a lot for your responses.
>
> I didn't know that you can explicitly refer to the timestamped
> snapshots of
> the artifacts. The limitation to the last 2 snapshots means that a
> push to
> flink-shaded can break our main CI? This sounds very fragile to
> me, given
> that the setup itself is probably a bit uncommon and difficult to
> understand.
>
> Maybe we should add an automated check to flink-shaded that warns
> if a PR
> would break Flink's CI? (by checking out flink and scanning the
> poms for
> references to a timestamp-to-be-deleted)
> Or we ask Infra to keep more than two snapshots for flink-shaded?
>
>
>
> On Mon, Apr 12, 2021 at 4:41 PM Chesnay Schepler
> <chesnay@apache.org <ma...@apache.org>> wrote:
>
> > a) yes.
> > b) maven by default adds a timestamp to snapshot artifacts that
> we can
> > use. The apache repository retains the last 2 snapshots, so we
> do need
> > to keep things in sync a fair bit, but there are rarely commits
> made in
> > flink-shaded that I don't think this will be a problem.
> > c) a-SNAPSHOT-uniquesuffix => a.0
> >
> > On 4/12/2021 3:07 PM, Robert Metzger wrote:
> > > Thanks a lot for your proposal, I'm generally open to the idea
> > >
> > > I have a few questions:
> > > a) Does this mean that we are changing flink-shaded to deploy
> snapshot
> > > artifacts to Apache's snapshot maven repository, and change
> Flink's
> > parent
> > > pom to point to this snapshot repo?
> > > b) How do you plan to generate the unique SNAPSHOT version on
> CI? Will we
> > > increment the version on every push to flink-shaded:master ?
> > > c) How do the unique SNAPSHOT versions relate to the final release
> > versions?
> > >
> > >
> > >
> > >
> > > On Mon, Apr 12, 2021 at 1:48 PM Konstantin Knauf
> <knaufk@apache.org <ma...@apache.org>>
> > wrote:
> > >
> > >> Sounds good. +1
> > >>
> > >> On Mon, Apr 12, 2021 at 1:23 PM Chesnay Schepler
> <chesnay@apache.org <ma...@apache.org>>
> > >> wrote:
> > >>
> > >>> Hello all,
> > >>>
> > >>> I would like to propose a change in how the Flink master
> interacts with
> > >>> Flink-shaded.
> > >>>
> > >>> TL;DR: Release snapshot artifacts for flink-shaded, and have
> the Flink
> > >>> master rely on specific snapshot versions for earlier
> dependency bumps.
> > >>>
> > >>>
> > >>> Aa a project we have come to the general conclusion that
> dependencies
> > >>> should be bumped as early in the release cycle as possible.
> This both
> > >>> prevents cases where some undefined amount of work is still
> waiting for
> > >>> as when we want to release the next version (working against
> the goal
> > of
> > >>> always being in a releasable state), and it gives us more
> time to
> > >>> evaluate the stability and performance of system. Finally it
> gives us
> > >>> ample time to look for alternatives if an issue is found.
> > >>>
> > >>> Currently, this conclusion is at odds with how we handle
> flink-shaded.
> > >>> Flink has always relied on flink-shaded artifacts that went
> through a
> > >>> proper release cycle. However, since we want to create as
> few releases
> > >>> as possible due to the overhead/noise/etc., flink-shaded
> releases are
> > >>> typically relegated to the end of the release cycle.
> > >>> This is particularly troublesome since flink-shaded
> dependencies are
> > >>> used in the core of Flink, and hence usage of them cannot be
> avoided.
> > >>>
> > >>> As a compromise between these 2 goals I propose the following:
> > >>> - we deploy SNAPSHOT artifacts for flink-shaded for every
> change made
> > >>> - every deployed artifact has a unique version, that is
> automatically
> > >>> set via maven (=> no overhead on our side)
> > >>> - once such an artifact is released we update the Flink
> dependency to
> > >>> point to this _specific_ flink-shaded snapshot artifact
> > >>> - to be clear, this is a manual step, which implies
> that things
> > >>> cannot break all of a sudden because something was pushed to
> > flink-shaded
> > >>> - once the Flink release cycle ends, we publish a proper
> flink-shaded
> > >>> release, and change the Flink dependency in the release branch
> > >> accordingly
> > >>> This should give us the best of both worlds: We have as few
> releases as
> > >>> necessary (at most 1 per Flink release cycle), but can
> update the
> > >>> dependencies in Flink as soon as possible.
> > >>> Furthermore, this can also be considered a test run for how
> multiple
> > >>> repos with the same release cycle could be developed in sync
> with each
> > >>> other.
> > >>>
> > >>> Let me know what you think.
> > >>>
> > >>> Regards,
> > >>>
> > >>> Chesnay
> > >>>
> > >>>
> > >> --
> > >>
> > >> Konstantin Knauf
> > >>
> > >> https://twitter.com/snntrable <https://twitter.com/snntrable>
> > >>
> > >> https://github.com/knaufk <https://github.com/knaufk>
> > >>
> >
> >
>
Re: [DISCUSS] Flink should rely on flink-shaded SNAPSHOT artifacts
Posted by Till Rohrmann <tr...@apache.org>.
Thanks for creating this proposal Chesnay. I do understand the problem you
want to fix.
What I am wondering is why we don't release flink-shaded more often. Does
the release process cause too much overhead? If this is the case, then we
could look into what is causing the overhead and whether we can improve the
situation. Concerning the noise, I personally don't see it as a problem.
My main concern is that it can easily slip our minds to change the
flink-shaded SNAPSHOT version to a non SNAPSHOT version and that it
introduces another manual step. If we forget to change the version, then
the Flink release does not build against a stable set of dependencies.
Moreover, I also second Robert's concern that a single commit to
flink-shaded can then break downstream projects (Flink in this case) if we
rely on the SNAPSHOT builds. Having to scan poms for some references sounds
like an indicator that this might not be the most straight forward approach.
Cheers,
Till
On Tue, Apr 13, 2021 at 9:26 AM Robert Metzger <rm...@apache.org> wrote:
> Thanks a lot for your responses.
>
> I didn't know that you can explicitly refer to the timestamped snapshots of
> the artifacts. The limitation to the last 2 snapshots means that a push to
> flink-shaded can break our main CI? This sounds very fragile to me, given
> that the setup itself is probably a bit uncommon and difficult to
> understand.
>
> Maybe we should add an automated check to flink-shaded that warns if a PR
> would break Flink's CI? (by checking out flink and scanning the poms for
> references to a timestamp-to-be-deleted)
> Or we ask Infra to keep more than two snapshots for flink-shaded?
>
>
>
> On Mon, Apr 12, 2021 at 4:41 PM Chesnay Schepler <ch...@apache.org>
> wrote:
>
> > a) yes.
> > b) maven by default adds a timestamp to snapshot artifacts that we can
> > use. The apache repository retains the last 2 snapshots, so we do need
> > to keep things in sync a fair bit, but there are rarely commits made in
> > flink-shaded that I don't think this will be a problem.
> > c) a-SNAPSHOT-uniquesuffix => a.0
> >
> > On 4/12/2021 3:07 PM, Robert Metzger wrote:
> > > Thanks a lot for your proposal, I'm generally open to the idea
> > >
> > > I have a few questions:
> > > a) Does this mean that we are changing flink-shaded to deploy snapshot
> > > artifacts to Apache's snapshot maven repository, and change Flink's
> > parent
> > > pom to point to this snapshot repo?
> > > b) How do you plan to generate the unique SNAPSHOT version on CI? Will
> we
> > > increment the version on every push to flink-shaded:master ?
> > > c) How do the unique SNAPSHOT versions relate to the final release
> > versions?
> > >
> > >
> > >
> > >
> > > On Mon, Apr 12, 2021 at 1:48 PM Konstantin Knauf <kn...@apache.org>
> > wrote:
> > >
> > >> Sounds good. +1
> > >>
> > >> On Mon, Apr 12, 2021 at 1:23 PM Chesnay Schepler <ch...@apache.org>
> > >> wrote:
> > >>
> > >>> Hello all,
> > >>>
> > >>> I would like to propose a change in how the Flink master interacts
> with
> > >>> Flink-shaded.
> > >>>
> > >>> TL;DR: Release snapshot artifacts for flink-shaded, and have the
> Flink
> > >>> master rely on specific snapshot versions for earlier dependency
> bumps.
> > >>>
> > >>>
> > >>> Aa a project we have come to the general conclusion that dependencies
> > >>> should be bumped as early in the release cycle as possible. This both
> > >>> prevents cases where some undefined amount of work is still waiting
> for
> > >>> as when we want to release the next version (working against the goal
> > of
> > >>> always being in a releasable state), and it gives us more time to
> > >>> evaluate the stability and performance of system. Finally it gives us
> > >>> ample time to look for alternatives if an issue is found.
> > >>>
> > >>> Currently, this conclusion is at odds with how we handle
> flink-shaded.
> > >>> Flink has always relied on flink-shaded artifacts that went through a
> > >>> proper release cycle. However, since we want to create as few
> releases
> > >>> as possible due to the overhead/noise/etc., flink-shaded releases are
> > >>> typically relegated to the end of the release cycle.
> > >>> This is particularly troublesome since flink-shaded dependencies are
> > >>> used in the core of Flink, and hence usage of them cannot be avoided.
> > >>>
> > >>> As a compromise between these 2 goals I propose the following:
> > >>> - we deploy SNAPSHOT artifacts for flink-shaded for every change made
> > >>> - every deployed artifact has a unique version, that is automatically
> > >>> set via maven (=> no overhead on our side)
> > >>> - once such an artifact is released we update the Flink dependency to
> > >>> point to this _specific_ flink-shaded snapshot artifact
> > >>> - to be clear, this is a manual step, which implies that things
> > >>> cannot break all of a sudden because something was pushed to
> > flink-shaded
> > >>> - once the Flink release cycle ends, we publish a proper flink-shaded
> > >>> release, and change the Flink dependency in the release branch
> > >> accordingly
> > >>> This should give us the best of both worlds: We have as few releases
> as
> > >>> necessary (at most 1 per Flink release cycle), but can update the
> > >>> dependencies in Flink as soon as possible.
> > >>> Furthermore, this can also be considered a test run for how multiple
> > >>> repos with the same release cycle could be developed in sync with
> each
> > >>> other.
> > >>>
> > >>> Let me know what you think.
> > >>>
> > >>> Regards,
> > >>>
> > >>> Chesnay
> > >>>
> > >>>
> > >> --
> > >>
> > >> Konstantin Knauf
> > >>
> > >> https://twitter.com/snntrable
> > >>
> > >> https://github.com/knaufk
> > >>
> >
> >
>
Re: [DISCUSS] Flink should rely on flink-shaded SNAPSHOT artifacts
Posted by Robert Metzger <rm...@apache.org>.
Thanks a lot for your responses.
I didn't know that you can explicitly refer to the timestamped snapshots of
the artifacts. The limitation to the last 2 snapshots means that a push to
flink-shaded can break our main CI? This sounds very fragile to me, given
that the setup itself is probably a bit uncommon and difficult to
understand.
Maybe we should add an automated check to flink-shaded that warns if a PR
would break Flink's CI? (by checking out flink and scanning the poms for
references to a timestamp-to-be-deleted)
Or we ask Infra to keep more than two snapshots for flink-shaded?
On Mon, Apr 12, 2021 at 4:41 PM Chesnay Schepler <ch...@apache.org> wrote:
> a) yes.
> b) maven by default adds a timestamp to snapshot artifacts that we can
> use. The apache repository retains the last 2 snapshots, so we do need
> to keep things in sync a fair bit, but there are rarely commits made in
> flink-shaded that I don't think this will be a problem.
> c) a-SNAPSHOT-uniquesuffix => a.0
>
> On 4/12/2021 3:07 PM, Robert Metzger wrote:
> > Thanks a lot for your proposal, I'm generally open to the idea
> >
> > I have a few questions:
> > a) Does this mean that we are changing flink-shaded to deploy snapshot
> > artifacts to Apache's snapshot maven repository, and change Flink's
> parent
> > pom to point to this snapshot repo?
> > b) How do you plan to generate the unique SNAPSHOT version on CI? Will we
> > increment the version on every push to flink-shaded:master ?
> > c) How do the unique SNAPSHOT versions relate to the final release
> versions?
> >
> >
> >
> >
> > On Mon, Apr 12, 2021 at 1:48 PM Konstantin Knauf <kn...@apache.org>
> wrote:
> >
> >> Sounds good. +1
> >>
> >> On Mon, Apr 12, 2021 at 1:23 PM Chesnay Schepler <ch...@apache.org>
> >> wrote:
> >>
> >>> Hello all,
> >>>
> >>> I would like to propose a change in how the Flink master interacts with
> >>> Flink-shaded.
> >>>
> >>> TL;DR: Release snapshot artifacts for flink-shaded, and have the Flink
> >>> master rely on specific snapshot versions for earlier dependency bumps.
> >>>
> >>>
> >>> Aa a project we have come to the general conclusion that dependencies
> >>> should be bumped as early in the release cycle as possible. This both
> >>> prevents cases where some undefined amount of work is still waiting for
> >>> as when we want to release the next version (working against the goal
> of
> >>> always being in a releasable state), and it gives us more time to
> >>> evaluate the stability and performance of system. Finally it gives us
> >>> ample time to look for alternatives if an issue is found.
> >>>
> >>> Currently, this conclusion is at odds with how we handle flink-shaded.
> >>> Flink has always relied on flink-shaded artifacts that went through a
> >>> proper release cycle. However, since we want to create as few releases
> >>> as possible due to the overhead/noise/etc., flink-shaded releases are
> >>> typically relegated to the end of the release cycle.
> >>> This is particularly troublesome since flink-shaded dependencies are
> >>> used in the core of Flink, and hence usage of them cannot be avoided.
> >>>
> >>> As a compromise between these 2 goals I propose the following:
> >>> - we deploy SNAPSHOT artifacts for flink-shaded for every change made
> >>> - every deployed artifact has a unique version, that is automatically
> >>> set via maven (=> no overhead on our side)
> >>> - once such an artifact is released we update the Flink dependency to
> >>> point to this _specific_ flink-shaded snapshot artifact
> >>> - to be clear, this is a manual step, which implies that things
> >>> cannot break all of a sudden because something was pushed to
> flink-shaded
> >>> - once the Flink release cycle ends, we publish a proper flink-shaded
> >>> release, and change the Flink dependency in the release branch
> >> accordingly
> >>> This should give us the best of both worlds: We have as few releases as
> >>> necessary (at most 1 per Flink release cycle), but can update the
> >>> dependencies in Flink as soon as possible.
> >>> Furthermore, this can also be considered a test run for how multiple
> >>> repos with the same release cycle could be developed in sync with each
> >>> other.
> >>>
> >>> Let me know what you think.
> >>>
> >>> Regards,
> >>>
> >>> Chesnay
> >>>
> >>>
> >> --
> >>
> >> Konstantin Knauf
> >>
> >> https://twitter.com/snntrable
> >>
> >> https://github.com/knaufk
> >>
>
>
Re: [DISCUSS] Flink should rely on flink-shaded SNAPSHOT artifacts
Posted by Chesnay Schepler <ch...@apache.org>.
a) yes.
b) maven by default adds a timestamp to snapshot artifacts that we can
use. The apache repository retains the last 2 snapshots, so we do need
to keep things in sync a fair bit, but there are rarely commits made in
flink-shaded that I don't think this will be a problem.
c) a-SNAPSHOT-uniquesuffix => a.0
On 4/12/2021 3:07 PM, Robert Metzger wrote:
> Thanks a lot for your proposal, I'm generally open to the idea
>
> I have a few questions:
> a) Does this mean that we are changing flink-shaded to deploy snapshot
> artifacts to Apache's snapshot maven repository, and change Flink's parent
> pom to point to this snapshot repo?
> b) How do you plan to generate the unique SNAPSHOT version on CI? Will we
> increment the version on every push to flink-shaded:master ?
> c) How do the unique SNAPSHOT versions relate to the final release versions?
>
>
>
>
> On Mon, Apr 12, 2021 at 1:48 PM Konstantin Knauf <kn...@apache.org> wrote:
>
>> Sounds good. +1
>>
>> On Mon, Apr 12, 2021 at 1:23 PM Chesnay Schepler <ch...@apache.org>
>> wrote:
>>
>>> Hello all,
>>>
>>> I would like to propose a change in how the Flink master interacts with
>>> Flink-shaded.
>>>
>>> TL;DR: Release snapshot artifacts for flink-shaded, and have the Flink
>>> master rely on specific snapshot versions for earlier dependency bumps.
>>>
>>>
>>> Aa a project we have come to the general conclusion that dependencies
>>> should be bumped as early in the release cycle as possible. This both
>>> prevents cases where some undefined amount of work is still waiting for
>>> as when we want to release the next version (working against the goal of
>>> always being in a releasable state), and it gives us more time to
>>> evaluate the stability and performance of system. Finally it gives us
>>> ample time to look for alternatives if an issue is found.
>>>
>>> Currently, this conclusion is at odds with how we handle flink-shaded.
>>> Flink has always relied on flink-shaded artifacts that went through a
>>> proper release cycle. However, since we want to create as few releases
>>> as possible due to the overhead/noise/etc., flink-shaded releases are
>>> typically relegated to the end of the release cycle.
>>> This is particularly troublesome since flink-shaded dependencies are
>>> used in the core of Flink, and hence usage of them cannot be avoided.
>>>
>>> As a compromise between these 2 goals I propose the following:
>>> - we deploy SNAPSHOT artifacts for flink-shaded for every change made
>>> - every deployed artifact has a unique version, that is automatically
>>> set via maven (=> no overhead on our side)
>>> - once such an artifact is released we update the Flink dependency to
>>> point to this _specific_ flink-shaded snapshot artifact
>>> - to be clear, this is a manual step, which implies that things
>>> cannot break all of a sudden because something was pushed to flink-shaded
>>> - once the Flink release cycle ends, we publish a proper flink-shaded
>>> release, and change the Flink dependency in the release branch
>> accordingly
>>> This should give us the best of both worlds: We have as few releases as
>>> necessary (at most 1 per Flink release cycle), but can update the
>>> dependencies in Flink as soon as possible.
>>> Furthermore, this can also be considered a test run for how multiple
>>> repos with the same release cycle could be developed in sync with each
>>> other.
>>>
>>> Let me know what you think.
>>>
>>> Regards,
>>>
>>> Chesnay
>>>
>>>
>> --
>>
>> Konstantin Knauf
>>
>> https://twitter.com/snntrable
>>
>> https://github.com/knaufk
>>
Re: [DISCUSS] Flink should rely on flink-shaded SNAPSHOT artifacts
Posted by Robert Metzger <rm...@apache.org>.
Thanks a lot for your proposal, I'm generally open to the idea
I have a few questions:
a) Does this mean that we are changing flink-shaded to deploy snapshot
artifacts to Apache's snapshot maven repository, and change Flink's parent
pom to point to this snapshot repo?
b) How do you plan to generate the unique SNAPSHOT version on CI? Will we
increment the version on every push to flink-shaded:master ?
c) How do the unique SNAPSHOT versions relate to the final release versions?
On Mon, Apr 12, 2021 at 1:48 PM Konstantin Knauf <kn...@apache.org> wrote:
> Sounds good. +1
>
> On Mon, Apr 12, 2021 at 1:23 PM Chesnay Schepler <ch...@apache.org>
> wrote:
>
> > Hello all,
> >
> > I would like to propose a change in how the Flink master interacts with
> > Flink-shaded.
> >
> > TL;DR: Release snapshot artifacts for flink-shaded, and have the Flink
> > master rely on specific snapshot versions for earlier dependency bumps.
> >
> >
> > Aa a project we have come to the general conclusion that dependencies
> > should be bumped as early in the release cycle as possible. This both
> > prevents cases where some undefined amount of work is still waiting for
> > as when we want to release the next version (working against the goal of
> > always being in a releasable state), and it gives us more time to
> > evaluate the stability and performance of system. Finally it gives us
> > ample time to look for alternatives if an issue is found.
> >
> > Currently, this conclusion is at odds with how we handle flink-shaded.
> > Flink has always relied on flink-shaded artifacts that went through a
> > proper release cycle. However, since we want to create as few releases
> > as possible due to the overhead/noise/etc., flink-shaded releases are
> > typically relegated to the end of the release cycle.
> > This is particularly troublesome since flink-shaded dependencies are
> > used in the core of Flink, and hence usage of them cannot be avoided.
> >
> > As a compromise between these 2 goals I propose the following:
> > - we deploy SNAPSHOT artifacts for flink-shaded for every change made
> > - every deployed artifact has a unique version, that is automatically
> > set via maven (=> no overhead on our side)
> > - once such an artifact is released we update the Flink dependency to
> > point to this _specific_ flink-shaded snapshot artifact
> > - to be clear, this is a manual step, which implies that things
> > cannot break all of a sudden because something was pushed to flink-shaded
> > - once the Flink release cycle ends, we publish a proper flink-shaded
> > release, and change the Flink dependency in the release branch
> accordingly
> >
> > This should give us the best of both worlds: We have as few releases as
> > necessary (at most 1 per Flink release cycle), but can update the
> > dependencies in Flink as soon as possible.
> > Furthermore, this can also be considered a test run for how multiple
> > repos with the same release cycle could be developed in sync with each
> > other.
> >
> > Let me know what you think.
> >
> > Regards,
> >
> > Chesnay
> >
> >
>
> --
>
> Konstantin Knauf
>
> https://twitter.com/snntrable
>
> https://github.com/knaufk
>
Re: [DISCUSS] Flink should rely on flink-shaded SNAPSHOT artifacts
Posted by Konstantin Knauf <kn...@apache.org>.
Sounds good. +1
On Mon, Apr 12, 2021 at 1:23 PM Chesnay Schepler <ch...@apache.org> wrote:
> Hello all,
>
> I would like to propose a change in how the Flink master interacts with
> Flink-shaded.
>
> TL;DR: Release snapshot artifacts for flink-shaded, and have the Flink
> master rely on specific snapshot versions for earlier dependency bumps.
>
>
> Aa a project we have come to the general conclusion that dependencies
> should be bumped as early in the release cycle as possible. This both
> prevents cases where some undefined amount of work is still waiting for
> as when we want to release the next version (working against the goal of
> always being in a releasable state), and it gives us more time to
> evaluate the stability and performance of system. Finally it gives us
> ample time to look for alternatives if an issue is found.
>
> Currently, this conclusion is at odds with how we handle flink-shaded.
> Flink has always relied on flink-shaded artifacts that went through a
> proper release cycle. However, since we want to create as few releases
> as possible due to the overhead/noise/etc., flink-shaded releases are
> typically relegated to the end of the release cycle.
> This is particularly troublesome since flink-shaded dependencies are
> used in the core of Flink, and hence usage of them cannot be avoided.
>
> As a compromise between these 2 goals I propose the following:
> - we deploy SNAPSHOT artifacts for flink-shaded for every change made
> - every deployed artifact has a unique version, that is automatically
> set via maven (=> no overhead on our side)
> - once such an artifact is released we update the Flink dependency to
> point to this _specific_ flink-shaded snapshot artifact
> - to be clear, this is a manual step, which implies that things
> cannot break all of a sudden because something was pushed to flink-shaded
> - once the Flink release cycle ends, we publish a proper flink-shaded
> release, and change the Flink dependency in the release branch accordingly
>
> This should give us the best of both worlds: We have as few releases as
> necessary (at most 1 per Flink release cycle), but can update the
> dependencies in Flink as soon as possible.
> Furthermore, this can also be considered a test run for how multiple
> repos with the same release cycle could be developed in sync with each
> other.
>
> Let me know what you think.
>
> Regards,
>
> Chesnay
>
>
--
Konstantin Knauf
https://twitter.com/snntrable
https://github.com/knaufk